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Executive  Summary 

The  Systems  Engineering  Research  Center  (SERC)  research  task  (RT-48)  focuses  on  a  Vision  held  by 
NAVAIR's  leadership  to  assess  the  technical  feasibility  of  creating/leveraging  a  more  holistic  Model- 
Based  Systems  Engineering  (MBSE)  approach.  The  expected  capability  of  such  an  approach  would 
enable  mission-based  analysis  and  engineering  that  reduces  the  typical  time  by  at  least  25  percent  from 
what  is  achieved  today  for  large-scale  air  vehicle  systems.  The  research  need  includes  the  evaluation  of 
emerging  system  design  through  computer  (i.e.,  digital)  models.  The  first  phase  of  the  effort  began 
investigating  the  technical  feasibility  of  moving  to  a  "complete"  model-driven  lifecycle  and  includes  four 
tasks  as  shown  in  Figure  1.  The  key  tasks  include: 


■  Task  1:  Surveying  Industry,  Government  and  Academia  to  understand  the  state-of  the-art  of  a 
holistic  approach  to  MBSE 

■  Task  2:  Develop  a  common  lexicon  for  MBSE,  including  model  types,  levels,  uses,  representation, 
visualizations,  etc. 

■  Task  3:  Model  the  "Vision,"  but  also  relate  it  to  the  "As  Is"  process 

■  Task  4:  Integrate  a  Risk  Management  framework  with  the  Vision 


1)  Global  scan  and  classification  of  holistic 
state-of-the-art  MBSE 


Use  discussion  framework  to  survey 
government,  industry  and  academia 
Quantify,  link 
and  trace  realized 
modeling 
capabilities 
to  Vision  (task  3) 


2)  Develop  Common  Lexicon  for  Model 
Levels,  Types,  Uses,  and  Representations 


Model  Types 


Structure/I  nterfaces 


Campaign 

Mission 

Engagement 

Engineering 


Behavior  (functions) 


Concurrency 


Resou  rces/En  vi  ronment 


IWC  J^rocess 


Address  two  classes  of 


Airworthiness  and 
Safety 

Program  Execution 


3)  Model  the  Vision  of  Everything  Done  with 
Modeis  and  Reiate  to  “As  is”  process 


4)  Fuiiy  integrate  modei-driven  Risk 
Management  and  Decision  Making 


Figure  1.  Four  Tasks  to  Assess  Technical  Feasibility  of  '"Doing  Everything  with  Models" 


The  NAVAIR  sponsor  envisioned  this  research  effort  would  take  approximately  two  years,  but  due  to  the 
ending  of  the  original  SERC  contract  in  December  of  2013,  this  first  phase  (Phase  I)  had  a  period  of 
performance  of  nine  months.  This  report  provides  details  about  each  task,  the  focus  of  the  research 
questions,  accomplishments,  and  the  plans  for  the  Phase  II  efforts,  which  are  to  continue  under  RT-118. 
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1  Introduction 


In  2013,  Naval  Air  Systems  Command  (NAVAIR)  at  the  Naval  Air  Station,  Patuxent  River,  Maryland 
initiated  a  research  task  (RT-48)  to  assess  the  technical  feasibility  of  creating/leveraging  a  more  holistic 
Model-Based  Systems  Engineering  (MBSE)  approach  to  support  Mission-based  Analysis  and  Engineering 
in  order  to  achieve  a  25%  reduction  in  development  time  from  that  of  the  traditional  large-scale 
weapon  systems.  The  research  need  included  the  evaluation  of  emerging  system  design  through 
computer  models.  The  first  phase  of  the  effort  created  a  strategy  and  began  collecting  and  structuring 
evidence  to  assess  the  technical  feasibility  of  moving  to  a  "complete"  model-driven  lifecycle. 

The  larger  context  of  the  NAVAIR  mission  seeks  a  Transformation  of  Systems  Engineering  (SE)  through 
MBSE,  where  MBSE^  is  used  in  the  most  general  way.  A  key  goal  is  to  leverage  virtual  designs  that 
integrate  with  existing  systems  data  and  simulations,  as  well  as  surrogates  at  varying  levels  of 
refinement  and  fidelity  to  support  a  more  continuous  approach  to  systems.  Family  of  Systems  (FoS), 
and  Systems  of  Systems  (SoS)  analysis  of  alternative  (AoA)  and  design  refinement.  Tradeoffs  in  this 
context  would  consider  potentially  non-optimal  solutions  that  can  close  the  gaps  most  rapidly  to 
support  the  warfighters  efforts  in  averting  new  or  emerging  threats.  This  should  allow  for  proposing 
solutions  to  the  mission/capability  gaps,  given  both  cost  and  time  constraints  and  tradeoffs,  but  be 
significantly  faster  than  using  the  current  process  while  satisfying  critically  important  safety  and 
airworthiness  requirements. 

The  Vision  of  NAVAIR  is  to  establish  an  environment  to  evaluate  the  emerging  system  design  through 
computer  models  and  demonstrate  system  compliance  to  user  performance  and  design  integrity 
requirements,  while  managing  airworthiness  risks.  It  is  anticipated  that  the  use  of  models  can 
streamline  or  radically  transform  the  decomposition  of  requirements  and  their  subsequent  integrated 
analysis,  which  is  currently  aligned  with  the  Department  of  Defense  (DoD)  systems  engineering  V- 
model.  By  providing  more  tightly  coupled  and  dynamic  linkage  between  the  two  sides  of  the  traditional 
"V,"  more  efficient  and  focused  requirements  decomposition  would  eliminate  thousands  of  pages  of 
documentation  delivered  via  contract  data  requirements  that  now  substitute  for  directly  invoking, 
manipulating,  and  examining  the  design  through  computer-based  models. 


1.1  Objective 

The  overarching  and  potentially  controversially  worded  research  question  is: 

■  Is  it  technically  feasible  to  "do  everything  with  models?"  (the  Vision) 

The  emphasis  by  the  sponsors  is  on  the  "technical  feasibility"  of  such  a  Vision.  It  is  acknowledged  that 
there  are  many  possible  hurdles  beyond  technical  feasibility  (e.g.,  organizational  adoption,  training, 
usability,  etc.),  but  they  have  in  general  been  reduced  in  priority  for  this  phase  of  the  effort. 

There  are  many  additional  research  questions  such  as: 

■  What  are  the  emerging  technologies  and  capabilities  that  will  enable  this  Vision? 

■  How  will  such  a  Vision  work  in  the  face  of  complex,  human-in-the-loop,  autonomous,  and 
adapting  system? 

■  Can  such  approaches  work  in  the  face  of  safety  and  airworthiness  requirements? 

■  What  are  the  technical  gaps  limiting  the  Vision? 


^  Some  use  the  term  Model  Based  Engineering  (MBE),  Model  Driven  Engineering  (MDE),  or  Model-Based  Systems 
Development  (MBSD). 
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■  What  are  the  approaches  to  deal  with  risk  when  making  decisions  based  on  virtual  models  and 
the  associated  simulations,  surrogates  and  analyses? 

There  are  four  key  tasks,  which  are  described  in  this  report,  but  a  key  decision  made  during  the  kickoff 
meeting  was  to  attempt  to  "model  the  Vision"  the  rationale  being  that: 

If  one  can  "do  everything  with  models"  then  we  should  be  able  to  "model  the  Vision." 

The  plan  is  to  look  to  Industry  (both  contract  developers  of  NAVAIR  systems  and  MBSE  technology 
providers).  Academia  and  Government  to  identify  the  most  advanced  holistic  state-of-the-art 
approaches  to  MBSE  and  represent  those  aspects  and  characteristics  in  the  "modeled  Vision."  There  are 
some  things  that  will  likely  be  challenging  to  model,  at  least  for  now  (e.g.,  human  cognitive  system 
interactions),  and  therefore  there  will  be  a  risk  framework  integrated  with  the  Vision.  This  risk 
framework  will  also  leverage  other  types  of  predictive  models  (e.g.,  stochastic)  to  both  embed  decision¬ 
making  knowledge  and  formalize  both  quantitative  and  qualitative  information  to  support  risk-informed 
decision-making. 


1.2  Accomplishments  in  Phase  1 

During  Phase  1,  the  objectives  for  the  research  needs  were  refined  through  meetings  and  working 
sessions  held  at  NAVAIR.  The  tasks  initiated  during  for  Phase  1  included: 

■  Task  1:  Surveying  Industry,  Government  and  Academia  to  understand  the  state-of  the-art  of  a 
holistic  approach  to  MBSE; 

o  Created  and  refined  a  discussion  collection  instrument,  associated  measurement  model,  and 
usage  guidelines  through  an  initial  set  of  discussions  with  industry,  government,  and 
academic  organizations 

o  The  discussion  instrument  identifies  those  indicators  of  capabilities  that  are  the  most  state- 
of-the-art,  which  align  with  NAVAIR's  Vision  (e.g.,  "crossing  the  virtual  V")  while  assessing 
risks  such  as  airworthiness  and  safety 

o  The  discussion  instrument  provides  a  way  to  transform  subjective  expert  judgments  into  a 
probabilistic  quantity  to  provide  a  means  for  assessing  the  technical  feasibility  of  the  Vision 

■  Task  2:  Develop  a  common  lexicon  for  MBSE,  including  model  types,  levels,  uses,  representation, 
etc. 

o  Created  a  lexicon  using  a  template-based  collection  and  website  generation  mechanism, 
with  visualization  for  continually  capturing  and  refining  the  lexicon  that  defines  terms 
related  to  models 

■  Task  3:  Model  the  Vision  and  relate  it  to  the  "As  Is"  process;  this  is  an  ongoing  process  to: 

o  Identify  "As  Is"  artifacts,  model  their  relationships  and  characterize  the  process  in  a  model 
o  Develop  mappings  from  the  "As  Is"  artifacts  to  model-based  candidates  for  the  Vision 
o  Created  a  "straw  man"  to  describe  the  concept  of  a  model-based  Vision 
o  Scoped  effort  to  Programs  of  Record 

■  Task  4:  Integrate  a  Risk  Management  framework  with  the  Vision; 

o  Identified  and  documented  the  strategies  being  researched  by  other  organizations  and 
identified  key  challenges  with  the  Vision  concept  of  "model  everything"  (e.g.,  dealing  with 
classes  of  problems  not  easy  to  model:  human  cognitive  properties) 


9 


1.3  Organization  of  Document 


Section  1  provides  a  statement  of  the  research  objectives,  overview  of  the  current  progress  and 
organization  of  this  report. 

Section  2  describes  the  approach  for  having  discussions  with  commercial,  government,  and  academic 
organizations  in  order  to  assess  the  most  holistic  state-of-the-art  use  or  vision  for  MBSE.  This  section 
provides  also  a  description  of  a  discussion  measurement  instrument  and  model  that  is  being  used  to 
transform  subjective  information  into  a  quantitative  representation. 

Section  3  describes  the  approach  for  developing  a  model  lexicon  to  characterize  such  things  as:  model 
levels,  model  types,  model  uses,  representations,  and  other  categories  related  to  "models." 

Section  4  discusses  the  scope  and  concept  of  the  Vision  model  and  its  relationship  to  the  "As  Is" 
artifacts  and  process  that  are  currently  in  place  for  developing  NAVAIR  air  vehicle  systems;  this  also 
includes  the  airworthiness  process. 

Section  5  discusses  a  framework  for  risk  identification  and  management  that  is  primarily  focused  on 
how  airworthiness  and  safety  risk  can  be  integrated  in  the  Vision  model,  but  it  will  also  deal  with 
program  execution  risks. 

Section  6  provides  some  conclusions  with  a  brief  summary  of  the  planned  next  steps. 

There  is  additional  backup  information,  including  a  list  of  acronyms  and  abbreviations  following  the 
conclusion. 
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2  Task  1  -  Assessing  the  State-of-the-Art  MBSE 


Commercial,  government,  and  academic  users  of  MBSE  have  seen  a  steady  rise  in  multi-level  and  multi- 
domain  model,  analysis  and  simulation  capabilities.  The  emerging  environments  support  varying 
degrees  of  fidelity  that  connect  to  different  and  complementary  views  of  the  system  under  analysis  and 
design.  The  rise  of  high-performance  computing  has  increased  modeling  and  simulation  capabilities  and 
made  large-scale  models  accessible. 

Our  team  developed  and  refined  a  guideline  for  our  collective  NAVAIR  team  to  hold  discussions  with 
Industry,  Government  and  Academia  (organizations)  to  understand  the  most  state-of-the-art  and 
holistic  approaches  to  MBSE.  The  objective  for  our  team  members  is  to  facilitate  conversations  through 
discussions  that  draw  out  insights  into  leading  advances  in  MBSE.  We  agreed  early  on  with  the  sponsor 
that  open-ended  discussions,  as  opposed  to  surveys,  would  bring  out  new  and  more  innovative  MBSE- 
related  approaches  and  strategies.  We  are  particularly  interested  in  demonstrations  of  actual  technical 
capabilities,  but  want  to  understand  the  critical  gaps  and  limitations  too. 

We  created  a  collection  instrument.  The  instrument  provides  a  constructive  approach  to  conduct  a 
discussion  with  organizations  as  well  as  a  way  to  provide  some  type  of  quantitative  measure  associated 
with  using  subjective  information  to  rate  the  "state-of-the-art"  of  a  holistic  approach  to  MBSE.  We  are 
using  a  qualitative  subjective  approach  that  computes  a  probabilistic  value  associated  with  crosscutting 
aspects  (factors)  associated  with  the  technical  Vision  for  this  task. 

The  collection  instrument  uses  an  Excel  spreadsheet  as  the  input  mechanism  to  collect  factor  values 
about  an  organization's  use  of  MBSE  as  discussed  in  Section  2.2.  Each  row  in  the  spreadsheet 
represents  the  subjective  information  associated  with  one  organization.  The  latest  version  of  the 
instrument  includes  one  organizational  classifier  and  22  factors.  We  have  tested  the  instrument  with 
eight  different  organizations  covering  different  domains  and  perspectives.  We  have  refined  the 
instrument  based  on  feedback  from  our  team's  use,  and  will  continue  to  do  so,  as  necessary,  as  we  have 
meetings  with  other  organizations. 

The  following  provides  additional  details  related  to  the  Task  1  effort,  process,  and  approach. 


2.1  Task  1  -  Process 

After  a  meeting  with  an  organization,  we  request  two  things  from  our  team  members  who  conducted 
the  discussion: 

■  Complete  one  row  of  the  spreadsheet;  see  Section  2.2  for  details  on  the  collection  process 

■  Write  a  short  summary  reflecting  on  the  key  unique  capabilities  of  the  organization 

The  spreadsheet  responses  will  be  incorporated  in  a  master  list.  The  value  for  each  factor  will  be 
entered  in  a  modeling  tool,  which  quantifies  the  subjective  inputs  provided  to  the  tool,  as  shown  Figure 
2.  The  maximum  value  of  the  mean  of  the  probability  distribution  is  100.  As  reflected  in  Figure  2,  it  was 
decided  that  because  there  are  some  organizations  that  require  confidentiality  or  proprietary 
information  agreements,  we  have  decided  to  keep  the  names  of  all  organizations  anonymous.  In 
addition,  a  narrative  will  be  created  for  each  organization;  this  narrative  will  highlight  the  most  key 
capabilities  and  challenges,  but  be  generalized  to  ensure  each  organization's  anonymity.  Additional 
details  about  interpreting  the  results  are  provided  in  Section  2.2. 
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Figure  2.  Collection  Instrument  Results 


2.2  Scenario  Collection 

Our  team  members  engage  in  the  discussions  with  organizations,  and  complete  the  spreadsheet 
collection  mechanism  as  shown  in  Figure  1  at  the  conclusion  of  the  meeting.  He/she  works  through  the 
row  and  uses  the  pull  down  menus  to  select  a  factor  value  of  Low,  Medium,  or  High  (see  Section  2.5.2 
for  details  on  Ranked  factor  values).  A  complete  list  of  factors  is  provided  in  a  worksheet  tab  of  the 
spreadsheet  collection  mechanism  titled:  Factor  Meaning-Definition.  Example  definitions  are  provided 
in  Section  2.2.3,  with  some  additional  rationale;  a  complete  set  of  definitions  is  provided  in  Discussion 
Collection  Instrument  Guide  and  provided  in  the  back  up  material  of  this  report. 

Team  members  may  want  to  use  one  spreadsheet  to  collect  all  of  the  discussions;  it  is  possible  and 
acceptable  that  after  a  few  meetings  with  organizations  that  one  or  more  of  the  factor  values  be 
changed  in  order  to  be  more  globally  consistent.  The  key  is  not  to  identify  a  particular  organization, 
rather  the  objective  is  to  identify  if  there  are  state-of-the-art  methods,  tools,  processes  and  innovative 
strategies  that  are  being  used  to  significantly  advance  the  development  and  deployment  of  systems 
through  MBSE  and  related  approaches,  and  to  incorporate  these  concepts  in  the  Vision  model  (see 
Section  4). 
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Organization  Name 
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Factors  (by  Category) 
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factors  value  (L,  M,  H) 

Figure  3.  Spreadsheet  Instrument  Collection 


2.2.1  Organizational  Type 

The  general  convention  used  is: 

■  Academia  -  this  should  include  strictly  academic  organizations;  other  organizations  performing 
research  should  either  be  Industry,  Commercial  or  Government 

■  Industry  -  these  are  organizations  that  are  using  MBSE  to  develop  products  and  systems  (e.g., 
those  contractors  to  NAVAIR  that  produce  air  vehicle  systems) 

■  Commercial  -  this  is  a  special  case  of  Industry  that  relates  to  MBSE  product  developers 

o  These  organizations  either  develop  MBSE  tools  and  services,  or  may  apply  them  with 
Industry  or  Government  customers 

o  These  organizations  are  in  the  list,  because  they  may  have  insights  into  some  of  the  leading 
or  novel  uses  of  the  tools,  and  they  are  aware  of  the  need  to  continually  advance  their  own 
product  and  services 

■  Government  -  this  includes  military,  and  other  non-military  organizations  such  as  Department  of 
Transportation,  and  the  FAA 


2.2.2  Organizational  Scope 

One  challenge  for  some  of  the  initial  uses  of  the  collection  mechanism  was  to  appropriately  reflect  on 
the  organization  scope  for  which  these  MBSE  usage  questions  apply.  Remembering  that  the  key 
objective  of  the  survey  is  to  assess  the  "Technical  Feasibility"  of  "doing  everything  with  model."  We 
recognize  that  actual  adoption  can  be  difficult,  and  might  not  make  sense  on  older  systems.  Therefore  it 
is  probably  best  to  hold  discussions  with  individuals  in  the  roles  such  Chief  Engineer,  Chief  Technical 
Offer,  Program  Manager  or  some  MBSE  technical  experts  in  the  organization.  To  carry  this  a  step 
further,  it  might  also  be  important  to  keep  the  "systems"  perspective  in  mind,  because  some  of  the 
concepts  discussed  may  have  been  applied  at  the  hardware  level  and  possibly  in  types  of  software  (e.g.. 
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the  control  laws  for  the  F35  are  built  in  Simulink,  with  auto  code  generation,  and  a  significant  portion  of 
auto  test  generation),  but  these  types  of  approaches  may  only  be  emerging  in  use  at  the  systems  level. 
We  seek  to  understand  how  comprehensive  the  use,  and  also  need  to  understand  the  technical  gaps. 
The  technical  gaps  areas  will  likely  need  to  have  additional  focus  related  to  risk  management  (Task  4). 

Finally,  this  research  is  not  limited  to  NAVAIR,  however  when  thinking  about  NAVAIR  systems  the  scope 
is  often  quite  large  and  involves  large-scale  multi-year  programs,  where  the  systems  are  actually  built 
by  one  or  more  contractors. 

Therefore,  we  would  like  to  know  the  organizational  scope  associated  with  the  MBSE  discussion: 
Program,  Project,  an  entire  Business  Unit,  Platform  (e.g.,  a  type  of  a  specific  aircraft,  tank,  automobile). 
Department,  or  Site. 


2.2.3  Factors  Definition  Example 

The  factor  categories  do  not  necessarily  relate  to  specific  MBSE  practices,  rather  they  are  higher-level 
characteristics  of  the  organization's  ability  to  leverage  the  use  of  models  and  the  associated 
technologies  that  enable  simulations  and  accelerate  the  analyses,  design,  synthesis,  V&V  and 
manufacturing  processes.  For  example: 

■  Crossing  the  Virtual  V  is  a  high-level  category  that  has  significant  weighting  in  the  model, 

because  our  sponsor  emphasized  this  as  a  critical  need  and  the  ability  to  understand  the  design 
capabilities  through  early  V&V  activities  at  the  system  and  mission  level  (as  opposed  to  the 
subsystem  or  component  level).  Therefore,  this  factor  category  has  three  main  factor 
characteristics: 
o  Simulation  of  Integration 

•  If  an  organization  has  simulations  of  integration  or  integrated  simulations  across 
domains  of  the  system,  and  especially  at  the  "higher"  levels  of  the  "V,"  this  is  a  likely 
indicator  that  such  an  organization  is  likely  to  have  the  ability  to  understand  simulations 
of  the  system  within  the  context  of  a  mission,  and  there  is  a  better  understanding  of  the 
integration  impacts,  because  the  simulations  are  integrated  or  represent  integration, 
including  critical  temporal  aspects  in  simulation 

•  This  includes  the  integration  of  surrogates,  use  of  instrumented  systems,  actual  system 
components,  new  prototypes,  and/or  in  development 

•  Other  attributes  of  this  type  of  simulation,  would  be  human-in-the-loop,  as  well  as 
multi-level  mixed  fidelity  simulations  that  provide  the  right  abstractions  at  the  right 
level 

o  Formal  analysis 

•  This  means  that  the  analysis  is  automated,  because  the  models  are  semantically  rich;  we 
are  looking  for  automated  analysis,  rather  than  looking  at  humans  performing  the 
analysis 

•  Models  are  increasingly  have  more  semantic  richness  that  enable  automated-types  of 
analysis,  and  models  are  increasingly  being  integrated  (see  factor  category  Cross 
Domain  Coverage) 

o  Domain  specific 

•  These  types  of  systems  involve  the  integration  of  many  disciplines 

•  Models  need  to  provide  the  relevant  abstractions  that  are  related  to  the  domain  of  the 
engineer  performing  the  work;  domain-specific  modeling  is  an  emerging  type  of 
modeling  that  often  provides  the  relevant  abstractions,  with  the  semantic  richness  to 
enable  automated  analysis,  simulation,  synthesis  (generation)  and  automated  test 
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•  DARPA-sponsored  research  that  demonstrated  the  capability  for  continuously  evolving 
Domain  Specific  Modeling  and  analyses  in  2008  as  an  emerging  capability  and  theme 
[16],  [33].  In  contrast,  modeling  languages  like  System  Modeling  Language  (SysML)  are 
general  purpose  [24]  they  generally  lack  the  semantic  richness  needed  for  formal 
analysis  leveraging  for  example  formal  methods  of  automated  V&V  [9];  while  they  may 
be  understood  by  system  engineers,  control  system  engineers  would  prefer 
Matlab/Simulink,  and  other  engineers  may  require  other  domain-specific  models  and 
tools  (e.g.,  computational  fluid  dynamics,  radio-frequency,  heat  transfer).  However, 
SysML  does  provide  an  underlying  framework  for  holding  system  model  information 
[36],  yet  the  models  are  not  executable  even  with  existing  plug-in  authoring  tools  [13] 
(see  Section  4.2  for  more  details). 


2.3  Organizations 

Table  1  shows  the  current  distributions  of  the  planned  discussions. 

Table  1.  Discussion  Groups  by  Category 


Discussion  Groups 

Category  (needs  discussion) 

Preliminary  Held 

Planned/ln  Process 

Commercial  -  provides  tools  and/ 
or  service  to  produce  systems 

3 

2 

Industry  -  produce  systems 

2 

8 

Government 

2 

6 

Academia/Consortium  Hybrid 

2 

2.4  Discussion  Summaries 

We  request  small  narratives  for  each  organization,  where  the  discussion  should  highlight  those 
organizational  uses  of  MBSE  that  represent  the  greatest  advances. 

For  example: 

An  industry  organization  provided  a  historical  perspective  on  evolutionary  adoption  of  MBSE.  Some 
of  the  more  exceptional  characterization  related  to  advances  they  have  made  that  included: 
composable  designs,  common  platforms,  catalogs  of  capabilities,  and  synthetic  representations  of 
the  environments.  Much  of  their  efforts  were  grown  out  of  SysML-centric  approaches.  They  are 
actively  using  model-based  metrics.  Live-info  for  simulation,  predictive  analytics  based  SysML, 
interpreting  and  tracing  data  consistency  across  disciplines,  data  set  metric,  trusted  models  for 
downstream  production,  RDM  consistency  and  semantic  links;  these  efforts  include  working  the 
MBSE  efforts  top-down  and  bottom-up,  starting  first  with  an  understanding  workflow,  and  meta- 
spec  concept. 

As  a  second  example,  we  talked  with  a  SME  from  the  Department  of  Transformation  that  has  been 
involved  in  crash  testing  of  automobiles  for  many  years.  The  focus  of  our  question  was  not  as  broad  as 
the  factors  covering  the  lifecycle  of  a  NAVAIR  program,  but  more  interested  in  the  question:  "are 
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automotive  companies  reducing  the  number  of  crash  tests  through  the  use  of  model?"  While,  we  did 
calculate  a  score  for  the  responses,  some  of  the  factors  may  not  directly  relate,  and  therefore,  we  are 
requesting  a  narrative  to  help  with  elevating  awareness  of  specific  uses  of  models  that  may  have 
relevance  to  the  overall  research  task.  There  is  a  factor  in  the  model  that  describes  Scope  Impact,  this 
value  is  used  to  characterize  the  scope  of  an  organization's  use  of  models. 


2.5  Predictive  Model 

This  section  is  provided  for  those  interested  in  more  details  about  the  mechanism  for  converting  the 
subjective  factors  into  a  quantitative  number.  The  model  is  created  using  a  Bayesian  Network  [32]  (BN) 
tool.  There  are  two  basic  reasons  we  selected  this  approach,  BNs: 

■  Provide  for  the  translation  of  subjective  information  into  quantitative  probabilities 

■  Allows  for  the  use  of  subjective  expert  qualitative  judgment  and  captures  the  casual  relationship 
between  subjective  factors 

The  outputs  are  also  probability  distributions,  which  means  that  they  provide  some  type  of  range  to 
provide  a  comparison  between  different  organizations.  The  specific  numbers  are  not  necessarily  as 
important  as  our  ability  to  compare  different  organizations  and  relate  the  responses  back  to  advanced 
uses  of  MBSE  and  related  enabling  technologies.  While  no  organization  may  have  all  "High"  values,  this 
approach  provides  a  way  to  look  at  relative  comparison  in  conjunction  with  the  narratives.  Each  of  the 
nodes  in  the  BN  shown  in  Figure  4  provides  a  type  of  weight  called  a  conditional  probability.  We  have 
used  the  team's  judgment  to  weight  the  different  nodes  in  a  way  that  would  relate  to  evaluating  the 
key  question  for  this  task:  is  it  technically  feasible  to  "do  everything  with  model."  In  addition,  we  will 
refine  the  weightings  as  we  proceed  through  discussions. 
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Figure  4.  Bayesian  Network  Underlying  Collection  Instrument 


2.5.1  Rationale  for  Bayesian  Networks 

A  Bayesian  network  is  a  representation,  which  organizes  one's  knowledge  about  a  particular  situation 
into  a  coherent  whole  [17].  They  are  increasingly  being  used  in  the  modeling  of  uncertain  and 
incomplete  knowledge.  Bayesian  thinking  is  inherently  more  intuitive  than  many  other  evaluation 
techniques;  it  best  reflects  commonsense  thinking  about  uncertainty  that  humans  have.  We  frequently 
use  words  like  ''likely,"  "rarely,"  and  "always"  to  express  varying  degrees  of  uncertainty.  Subjective 
probability  is  our  way  of  assigning  numbers  (between  0  and  1)  to  these  different  degrees  of  uncertainty, 
and  our  probabilities  can  change  as  we  are  presented  with  new  information,  or  we  have  new 
experiences  which  cause  a  shift  in  beliefs  or  expectations.  When  this  shift  occurs,  the  way  our 
probabilities  change  are  governed  by  Bayes'  rule. 

A  Bayesian  network,  as  used  in  this  framework,  is  a  joint  probability  distribution  and  as  such,  any 
question  that  can  be  asked  in  a  probabilistic  form  can  be  answered  with  a  stated  level  of  confidence. 
Some  typical  questions  might  be: 

■  Given  a  set  of  effects,  what  are  the  causes? 

■  How  can  an  outcome  be  controlled,  given  a  set  of  circumstantial  values? 

■  If  we  model  a  causal  relationship,  what  result  would  an  intervention  or  change  bring? 

While  there  are  several  ways  to  structure  a  Bayesian  network,  we  used  prior  experience  to  structure  the 
model.  The  subjective  factors  in  the  spreadsheet  instrument  map  directly  to  the  yellow  oval  nodes  of 
the  BN  model.  The  purple  rectangles  are  intermediate  nodes  and  generally  relate  to  factor  categories. 
The  orange  rectangles  represent  the  probability  outputs  of  both  Technical  state  of  the  art  (Task  3)  and 
the  Technical  Risk  state  of  the  art  (Task  4). 
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2.5.2  Data  -  Likert  Scales  (Ranked  Scales) 


The  subjective  factors  in  the  model  use  a  Ranked  node  type,  which  is  a  type  of  Likert  Scale.  It  is 
important  to  note  that  although  Likert  scales  are  arbitrary,  they  can  retain  a  level  of  reliability  for  our 
use.  The  value  assigned  to  a  Likert  item  has  no  objective  numerical  basis,  either  in  terms  of  measure 
theory  or  scale  (from  which  a  distance  metric  can  be  determined).  In  this  case,  the  value  assigned  to  a 
Likert  item  has  been  determined  by  the  researcher  constructing  the  Bayesian  network,  but  can  be 
refined  as  the  research  progresses.  The  results  have  been  a  balanced  representation  of  strata  and 
detail. 

Typically,  Likert  items  tend  to  be  assigned  progressive  positive  integer  values.  Likert  scales  usually  range 
from  2  to  10  -  with  5  or  7  being  the  most  common.  In  this  model,  3  levels  are  used,  at  least  for  now  as  it 
minimizes  the  number  of  computational  states,  which  minimizes  time  for  the  analysis.  The  progressive 
structure  of  a  Likert  scale  is  such  that  each  successive  Likert  item  is  treated  as  indicating  a  'better' 
response  than  the  preceding  value.  Note  that  the  direction  of  'better'  (i.e..  Higher)  depends  on  the 
wording  of  the  factor  definition,  which  is  provided  in  Section  2.2.3. 

In  terms  of  good  practice,  a  bias  in  the  computations  may  result  if  the  suppliers  of  data  for  the 
framework  do  not  agree  on  the  relative  values  of  each  factor.  However,  there  are  enough  factors  that  a 
bias  in  a  one  or  two  values  will  likely  not  skew  the  results  significantly. 
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3  Task  2  -  Common  Model  Lexicon 


The  team  realized  early  the  need  for  a  common  lexicon  when  discussing  modeling  in  the  systems 
engineering  domain,  and  in  fact,  in  the  broader  engineering  space.  An  example  of  this  is  what  is  meant 
by  the  word  "model."  Most  engineers  will  agree  that  a  model  is  a  facsimile  of  reality.  Yet,  to  an 
industrial  engineer,  a  model  may  represent  a  production  facility;  to  a  mechanical  engineer  it  may  be  a 
finite  element  model  analysis;  to  a  systems  engineer  it  may  be  an  IDEFO  or  a  SysML  representation  of 
the  system,  subsystem,  or  some  lower  level  element.  None  of  those  perspectives  are  wrong;  they  are 
just  different  views  of  some  part  of  the  same  enterprise. 

Therefore,  a  task  was  established  to  define  terms  used  across  the  engineering  modeling  landscape. 
Some  claim  that  there  is  no  existing  model  lexicon  or  taxonomy  [2],  although  there  are  a  number  of 
different  types  of  taxonomies  that  all  fit  within  the  more  general  context  of  a  model  lexicon  [14],  [36]. 
The  Object  Management  Group  (OMG)  in  conjunction  with  INCOSE  has  established  an  Ontology  Action 
Team  to  work  on  similar  efforts  [30].  The  NDIA  Modeling  &  Simulation  Committee  is  about  to  approve 
the  Final  Report  on  the  Identification  of  Modeling  and  Simulation  Capabilities  by  Acquisition  Life  Cycle 
Phase  [3].  For  NAVAIR,  this  was  an  immediate  need  for  our  research  task  and  we  were  tasked  with 
developing  a  common  lexicon. 


3.1  Ontology  vs.  Lexicon 

According  to  Wikipedia,  ontologies  are  the  structural  frameworks  for  organizing  information  and  are 
used  in  artificial  intelligence,  the  Semantic  Web,  systems  engineering,  software  engineering,  biomedical 
informatics,  library  science,  enterprise  bookmarking,  and  information  architecture  as  a  form  of 
knowledge  representation  about  the  world  or  some  part  of  it  [35].  The  creation  of  domain  ontologies  is 
also  fundamental  to  the  definition  and  use  of  an  enterprise  architecture  framework. 

A  lexicon  is  a  similar  concept  -  it  is  normally  a  book  or  glossary  like  document,  or  words  (and  their 
definitions)  in  a  language  or  domain,  arranged  in  alphabetical  order.  The  team  decided  that  a  simple 
glossary  would  not  be  sufficient  because  it  does  not  show  the  relationships  between  terms. 

In  simplistic  terms,  an  ontology  becomes  a  complex  network  of  words,  and  their  relationships  to  each 
other.  A  lexicon  is  a  glossary.  Neither  was  exactly  what  was  needed  for  this  project.  Instead  a  hybrid  is 
needed.  The  team  needs  something  that  provides  definitions  and  simple  relationships  -  not  complex, 
rigid  definitions.  We  chose  to  use  the  word  Lexicon,  though  the  words  could  also  be  represented  in  a 
tree-like  structure  that  is  common  for  ontologies. 


3.2  Tool  for  Representing  Word  Relationships 

There  are  tools  available  for  creating  ontologies.  There  actually  exists  a  class  of  workers  that  consider 
themselves  Ontologists.  These  tools  come  in  many  different  flavors  -  from  open  source  tools  to 
commercial  tools.  The  common  thread  is  that  they  create  graphical  representations  as  shown  in  an 
example  in  Figure  5.  These  tools  require  rather  rigorous  definitions  and  relationships  to  complete.  The 
open  source  tools  are  actually  very  good,  and  very  robust.  However,  after  some  evaluation  of  available 
open  source  tools,  the  team  decided  that  it  would  be  better  to  create  a  straightforward  spreadsheet  of 
terms  (e.g.  a  Lexicon),  and  then  create  a  script  that  could  represent  that  lexicon  graphically. 
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Figure  5.  Sample  Graphic  Representation  from  Ontological  Software 


3.3  The  Lexicon 

A  spreadsheet  was  first  created  in  Excel.  At  first,  the  team  was  simply  capturing  the  words,  their 
definition,  and  where  it  made  sense,  a  key  reference  or  two  for  that  definition.  Table  2  shows  the 
implementation  of  this  data  gathering  spreadsheet.  Once  the  decision  was  made  to  create  a  tool  to 
make  this  information  available  graphically,  and  also  on  the  web,  it  became  apparent  that  a 
"relationship"  data  element  was  necessary.  Therefore,  the  data  collection  tool  captures: 

■  Name 

■  Has  Parents  [0  or  more]  separate  with  if  more  than  one 

■  Definition 

■  Sample  Usage 

■  Also  Known  As 

■  Key  URL  (optional) 

The  current  spreadsheet  represents  a  continuous  accumulation  of  relevant  terms,  their  definitions,  and 
their  classification.  The  initial  definitions  have  been  drawn  from  readily  available  sources  on  the 
Internet  (often  from  Wikipedia  where  the  assumption  is  that  it  has  been  created  by  a  group  of  people 
with  both  knowledge  and  passion  about  the  subject).  In  other  cases  members  of  the  research  team 
have  authored  a  definition  based  on  their  understanding  of  the  term  in  a  relevant  context.  The  team  is 
using  the  spreadsheet  feature  of  GoogleDocs  to  foster  a  collaborative  effort. 
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Table  2.  Initial  Lexicon  Capture  Tool 


Term 

Definition 

References 

MBSE 

Model-based  systems  engineering  (MBSE)  is  the  formalized 
application  of  modeling  to  support  system  requirements,  design, 
analysis,  verification,  and  validation  activities  beginning  in  the 
conceptual  design  phase  and  continuing  throughout  development 
and  later  life  cycle  phases. 

(INCOSE,  Systems  Engineering  Vision  2020,  Version 
2.03,  TP-2004-004-02,  September  2007) 

MBSD 

An  engineering  approach  that  promotes  the  use  of  models  to 
develop,  integrate  and  monitor  systems  across  disciplines, 
environments  and  scenarios. 

MBE 

Model  Based  Engineering  -  Model-Based  Engineering  (MBE)  as  an 
approach  to  engineering  that  uses  models  as  an  integral  part  of  the 
technical  baseline  that  includes  the  requirements,  analysis,  design, 
implementation,  and  verification  of  a  capability,  system,  and/or 
product  throughout  the  acquisition  life  cycle. 

(NDIA  -  Final  Report  of  the  MBE  Subcommittee) 

Concurrent  Engineering 

reducing  time  between  project  launch  and  delivery  by  operating 
elements  of  the  process  in  parallel.  Concurrent  engineering  is 
simply  a  practice  of  doing  as  much  of  the  system  life  cycle  design  and 
development  in  parallel  as  possible  with  the  inevitable  rework 
associated  with  the  risk  of  parallel  development  -  as  you  get  into 
more  detail  that  will  reveal  findings  that  ripple  back  across  the 
'already'  completed  work  since  you  are  trying  to  tackle  everything  at 

once 

Intuitively,  many  of  the  terms  in  this  spreadsheet  are  ambiguous  and  their  meaning  is  highly  dependent 
on  the  context  and  usage  domain.  This  has  been  found  to  be  true  in  reality  also  as  terms  are  collected 
from  various  domains.  It  is  therefore  important  to  emphasize  that  this  is  an  evolving  process. 


3.4  Sources  of  Information 

There  were  a  number  of  sources  used  for  this  initial  Lexicon.  Journal  papers  on  MBSE  provided  a  good 
first  cut.  Interestingly,  an  article  from  the  Journal  of  Object  Technology  [21]  proved  to  be  very  useful. 
Other  sources  included  The  Open  Group,  the  Object  Management  Group,  INCOSE,  NDIA,  and  Wikipedia. 


3.5  Web  Presentation 

A  short  script  was  created  that  takes  the  information  contained  in  the  data-entry  spreadsheet,  and 
publish  the  results  to  the  web.  Figure  2  shows  the  published  page  as  it  looks  at  the  time  of  this  report, 
(http://www.markblackburn.com/MBSE/modelJexicon.html)^.  This  page  includes  four  sections: 

■  Model  Lexicon  Overview  (Figure  6) 

■  Model  Representation/Lexicon  (Figure  8) 

o  This  is  a  generated  image  produced  by  vizGraph,  but  with  over  300  lexicon  items  it  is 
difficult  to  use,  although  it  reflects  the  interrelationships  of  the  lexicon  elements 

■  Hyperlinked  Tree  of  the  Model  Lexicon  (Figure  7) 

o  As  an  alternative,  a  collapsible  and  expandable  tree  (outline)  allows  people  to  understand 
the  hierarchy  of  model  lexicon  with  hyperlinks  to  a  particular  lexicon  definition. 

■  Definitions  -  A  common  structure  is  used  for  each  term  (Figure  9) 


^  The  final  location  of  the  lexicon  may  move  to  another  location. 
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Figure  6.  Published  Web  Page  from  Data  Collection  Spreadsheet 


Model  Representation 

There  is  a  graphical  representation  of  the  lexicon  generated  by  \izGraph. 
Click  here  image. 

Note:  this  is  a  large  image  and  make  take  a  few  seconds  to  load. 

Model  Lexicon 

Model  Lexicon  Tree 

Model  lexicon 

Concurrent  engineering 

+  Definition 
+  Model 

+  Model  acquisition 
+  Model  levels 
“1“  Model  management 
+  Model  representations 
+  Model  types 
+  Model  uses 
+  Modeling  approach 
+  Modeling  standards 
+  System 


Figure  7.  Model  Representation  and  Lexicon  Tree 
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Figure  8.  Partial  Graphical  Representation  of  Lexicon 

The  definitions  table  shown  in  Figure  9,  is  a  screen  image  from  the  website,  and  includes  the  following 
columns: 

■  Name 

■  Definition 

■  Parent 

o  This  is  a  hyperlink  to  the  parent  in  the  table 

■  Tree 

o  This  is  a  hyperlink  back  to  the  collapsible  and  expandable  tree  (outline);  clicking  on  this 
hyperlink  takes  the  focus  back  to  the  name  in  the  tree  only  if  the  item  is  expanded  in  the 
tree 

■  Sample  Use 

■  Key  Source  (if  applicable) 
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Model  Lexicon  Definitions 


Name 

Definition 

Parent 

Tree 

Sample  Use 

Source 

2d  modeling: 

a  geometric  model  of  an  object 
as  two-dimensional  figure, 
usually  on  the  Euclidean  or 
Cartesian  plane 

mechanical 

modcline 

mechanical 
modeling.  2d 
modclinn 

3d  solid  model: 

the  product  of  3D  solid 
modeling 

mechanical 

modeling 

mechanical 
modeling.  3d 
solid  model 

3d  solid 
modeling: 

is  the  process  of  developing  a 
mathematical  representation  of 
any  three-dimensional  surface 
of  object  (cither  inanimate  or 
living)  via  specialized  software 

mechanical 

modeling 

mechanical 
modeling.  M 
sohd  modeline 

More  bample  _ ^ _ 

Model  driven 
architectare: 

a  software  design  approach  for 
the  development  of  software 
systems 

model 

technique 

mooci 
technique, 
model  driven 

architecture 

http:/Avww.om2.ore/mda 

Model  levels: 

Level  of  the  system  or  system 
of  systems;  Also  discussed  in 
terms  of  Resolution  level.  The 
amount  of  detail  or  degree  of 
aggregation  employed  in  the 
model  or  simulation 

model  lexicon 

model  lexicon. 

model  levels 

Often  used  in  modeling 
and  simulation  world  to 
discuss  the  differen  types 
of  models 

Model  lexicon: 

the  words  used  in  a  language  or 
by  a  person  or  group  of  people 

model  lexicon 

This  is  a  lexicon 
associated  with  terms 
derived  fix)m  models  and 
modeling 

Model 

management: 

Approaches  to  managing 
models 

model  lexicon 

model  lexicon. 

model 

manacement 

Configuration  control, 
PLM,  CATIA 

Figure  9.  Tabular  Representation  of  Lexicon 


3.6  Recommendations  Moving  Forward 

1.  We  expect  as  the  effort  continues,  team  members  will  continue  to  collaborate  in  the  definition 
and  classification,  causing  discussion  related  to  their  relevance  and  "correctness." 

2.  Additionally,  the  intent  is  that  the  broader  community  will  contribute  examples  and  sample 
usages  of  the  terms  to  improve  the  understanding  and  proper  use  in  various  contexts. 

3.  We  will  therefore  provide  mechanisms  that  allow  for  feedback/annotation  from  the  community 
and  a  basic  change  control  process. 

4.  It  might  be  good  to  add  a  "comment"  link  on  each  table  row  on  the  website  that  could  link 
directly  to  the  corresponding  row  in  the  Google  spreadsheet  to  enable  the  submission  of  a  new 
terms  and  definitions  directly  into  the  spreadsheet  (or  database). 

5.  A  longer-term  plan  would  be  to  drive  the  graphical  image,  and  textual  listing  from  a  database 
instead  of  a  spreadsheet. 
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4  Task  3  -  Modeling  the  Vision  and  Relating  to  the  "As  Is" 


The  concept  of  the  Vision  model  is  not  a  representation  of  a  NAVAIR  air  vehicle  system,  rather  the 
Vision  model  must  include  the  required  information  (data)  and  embedded  knowledge  that  is  normally 
captured  in  documents,  drawing,  specifications,  pseudo-formal  models,  and  tests  (some  refer  to  it  as 
the  "total"  system  model  [37]).  This  includes  or  subsumes  every  piece  of  information  that  relates  to  the 
artifacts  captured  in  the  "As  Is"  process,  but  should  also  include  formalized  information  such  as  the 
inputs  and  outputs  of  modeling  and  simulations,  analyses,  surrogates,  risk  information,  etc.  and  include 
specific  versions  of  each  tool,  simulation,  and  analysis  engine  used  to  support  the  necessary  evidence 
required  to  produce  an  airworthy  system  version.  Ideally,  this  should  include  every  piece  of  information 
to  the  Bill  of  Material  (BOM),  including  models  to  manufacturing  and  models  to  training. 

Preliminary  discussions  with  organizations  suggest  that  some  individuals  and  organizations  understand 
the  Vision  model  concept.  Some  are  attempting  to  develop  variants  on  the  concept  that  are  more 
specific  to  product  development.  Some  have  cross-business/discipline  projects  established  to  refine 
strategies  to  roll  out  and  support  adoption  by  programs  in  these  different  business  units.  Other  efforts 
are  focused  more  at  the  software  level  (using  the  characterization  Model  Driven  Engineering  [MDE]) 
[23].  One  study  cited  a  multi-level,  multi-domain  instance  case  that  started  at  the  campaign  level 
moving  down  through  the  mission,  engagement,  and  engineering  levels  [1].  There  are  also  organizations 
that  claim  to  be  applying  MBSE,  yet  they  have  not  seen  the  benefits;  we  understand  that  there  are 
often  adoption  challenges  [10],  and  that  is  why  our  sponsor  has  directed  us  to  focus  on  the  technical 
feasibility  for  this  phase  of  the  research. 

While  we  are  just  beginning  to  have  discussions  with  organizations,  it  is  a  fairly  consistent  message  that 
many  organizations  have  not  defined  a  Vision  model.  Instead  they  are  involved  in  an  evolutionary 
process  of  model  adoption,  and  many  want  to  better  understand  the  return  on  investment.  We  don't 
know  some  of  the  specific  details  yet  as  several  of  the  proprietary  discussions  are  in  the  coordinating 
stages  (see  Task  1).  Key  to  NAVAIR  is  that  we  also  do  not  know  how  well  these  efforts  address  some 
critically  important  requirements  for  NAVAIR  such  as  airworthiness  and  safety.  In  addition,  some  of 
these  organizations  are  working  on  a  subset  of  the  problem  (e.g.,  V&V)  [24],  while  others  are 
approaching  this  from  the  contractor  point-of-view,  which  is  significantly  different  from  that  of  NAVAIR. 
NAVAIR  is  working  in  the  early  stages  of  DoD  5000.02  [19]  lifecycle  (i.e..  Milestone  A,  B,  C),  and  they 
ultimately  produce  requirements  and  design  constraints  that  are  provided  to  the  contractors.  There  is  a 
significant  amount  of  research  needed  on  this  task.  Our  approach  is  to  work  this  effort  in  conjunction 
with  our  state-of-the-art  discussions  (Task  1). 

The  objective  for  the  Vision  should  address  the  questions: 

■  Can  we  create  models  to  cover  every  type  of  artifact  that  is  required  to  produce  a  system  and 
comply  with  DoD  and  NAVAIR  processes  and  requirements  (e.g..  Airworthiness)? 

■  Can  we  use  model-based  simulation,  analysis,  synthesis  and  generation  to  rapidly  traverse  the 
"Virtual  Vee"  and  continuously,  both  horizontally  and  vertically,  leverage  evolving  digital 
representations  (e.g.,  models,  surrogates)  to  assess  the  system  design  at  various  levels  of  fidelity 
in  the  context  of  continuously  evolving  mission  scenarios? 

■  How  does  the  risk  framework  fit  into  the  model? 

■  What  would  a  "Model-Driven"  process  be?  With  model-driven  processes  there  are  many  types 
of  automated  workflows  engines  that  automate  the  processes  inherent  in  producing  and 
relating  model-based  artifacts  (see  Section  4.5.1  for  an  example) 
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4.1  Context  for  Relating  "As  Is"  Artifacts  and  Process  to  Vision 

We  initially  developed  (as  a  straw  man,  see  Section  4.3)  an  example  model  in  System  Modeling 
Language  (SysML)  that  represented  the  Integrated  Warfighter  Capability  (IWC).  The  example  provided  a 
common  understanding  that  the  goal  of  the  modeled  Vision  is  going  to  formally  characterize  all  of  the 
"data,"  relationships,  automation  throughout  the  entire  lifecycle,  including  for  example  the  relationship 
to  data  used  by,  and  produced  by  modeling  and  simulation,  analyses  and  other  resources,  as  well  as 
evidence  captured  within  the  models  to  support  risk  assessment  and  management  (see  Section  5).^ 

From  a  high-level  perspective,  as  reflected  in  Figure  10,  Task  3  is  a  collaborative  effort  being  worked  by 
our  SERC  team,  SMEs  from  NAVAIR,  Naval  Post  Graduate  School  (NPS),  MITRE,  and  consultants  who 
have  extensive  NAVAIR  and  aircraft  system  engineering  experience.  This  section  provides  a  summary  of 
our  Phase  I  efforts.  The  following  enumerates  subtasks  for  Task  3  (the  list  order  is  aligned  with  the 
elements  in  Figure  10): 

1.  Developing  a  CORE'*  model  representation  of  a  derived  list  of  artifacts  that  are  currently 
produced  to  support  NAVAIR  System  Engineering  Technical  Review  (SETR)  process  (see  Section 
4.5  for  details) 

o  It  is  important  to  understand  the  artifacts  that  are  produced  to  comply  with  the  "As  Is" 
process,  along  with  the  relationship  and  dependencies  among  these  artifacts 
o  In  the  Vision,  the  information  described  in  these  artifacts  (some  of  which  are  models  today) 
must  be  ultimately  represented  in  models  (digital  form),  or  be  derivable  from  models 

2.  Representation  of  the  "As  Is"  process,  which  relates  to  the  DoD  5000.02  and  SETR  process 
o  The  analysis  of  the  "As  Is"  artifacts  and  process  should  provide  a  means  to  assess  the 

completeness  of  the  Vision,  and  help  people  understand  how  a  process  would  work  when 
transitioning  from  a  document-centric  operational  model  to  a  model-centric  approach 
o  As  we  are  looking  to  leverage  existing  results,  in  our  first  working  session  it  was  suggested 
that  we  look  at  the  Acquisition  Guidance  Model  (AGM)  developed  by  MITRE  [15]  (see 
Section  4.4  for  details) 

3.  Development  of  a  representation  of  the  Airworthiness  Process^ 

o  This  effort  will  characterize  those  critical  aspects  that  make  the  NAVAIR  requirements  more 
challenging  than  for  other  organizations 

o  The  types  of  required  Airworthiness  evidence  (e.g..  Flight  clearance)  must  be  identified  and 
presented  either  in  some  model  representation  (4)  and/or  support  risk-based  decision 
making,  which  should  be  captured  in  conjunction  with  the  Vision  (5) 

4.  Model  representation  that  subsumes  all  information  that  is  currently  represented  in  the  "As  Is" 
process,  if  deemed  to  be  necessary,  and  all  of  the  associated  digitized  automation  that  is 
required  to  transform  the  process 

o  This  will  be  significantly  informed  by  the  discussions  (Task  1) 

5.  The  integrated  risk  framework  (see  Section  5  for  details) 

6.  The  associated  process  for  apply  the  Vision  model;  in  many  instances,  when  the  information  is 
formalized  in  model,  a  corresponding  model-driven  automated  workflow  is  also  automated, 
however,  because  of  the  aspects  of  risk  and  airworthiness,  it  is  likely  that  there  are  some 
human-driven  steps  in  the  process 


^  There  are  number  of  useful  representations  and  documentation  that  are  not  currently  released  for  public 
viewing. 

'*  We  are  not  promoting  any  specific  modeling  tool. 

^  This  effort  was  started  in  our  February  working  session  and  is  being  supported  by  our  MITRE  team  partner. 


27 


Pre-MDD 

Material 
Solution  I 

Analysis  J 

Technology 

Development 

Engineering  & 
Manufacturing 
Development 

Production  & 
Deployment 

1)  CORE  Representation  of 
Artifacts  and  Relationships 
(Inherent  Process) 


5)  Risk-based  Decision 
Making 

4)  Model  Representation  of  Artifacts 
and  Associated  Model-Based 
Automation  with  Integrated  Surrogates 
(Simulation,  Analysis,  Synthesis,  etc.) 


Figure  10.  Model  Vision  at  Program  of  Record  Scope  and  Integrate  Risk-based  Decision  Framework 

The  following  subsections  are  presented  primarily  in  chronological  order  associated  with  research 
investigation,  working  sessions,  and  task  scoping  and  refinement. 


4.2  Modeling  AND  Tools  FOR  THE  Vision 

Our  team  has  had  numerous  discussions  about  modeling  representations,  languages  and  tools  for  the 
Vision.  The  examples  in  Section  4.3  use  SysML,  which  is  a  standard  modeling  language.  It  is  general  [24], 
but  there  are  limitations.  The  basic  SysML  diagrams  in  the  modeling  environments  are  mostly  static. 
System  engineering  models  defined  in  SysML  are  descriptive  in  nature  and  do  not  directly  produce 
analytical  results  [24],  nor  are  they  executable  [13].  Different  tool  vendors  provide  extensions  or  their 
own  analytical  capabilities  that  solve  SysML  parametric  diagram  [11].  Since  the  parametric  relationships 
are  solved  as  a  system  of  equations,  the  analytical  model  is  limited  to  simple  equations.  To  be  able  to 
use  more  sophisticated  engineering  analyses,  external  analysis  tools  need  to  be  connected  to  SysML 
models.  Other  research  efforts  are  attempting  to  leverage  other  standard  modeling  languages  such  as 
Modelica  that  have  a  broad  range  of  analytical  support  through  integration  between  SysML  and  the 
Modelica  [31].  Domain  Specific  Modeling  environments  (e.g.,  Simulink  for  control  systems)  often  have 
richer  semantics  (e.g.,  structure,  behavioral  and  sometimes  temporal)  to  support  dynamic  analyses, 
simulation  and  some  have  formal  method  analysis  and  automated  test  generation  [9],  [33].  Other 
approaches  provide  process  integration  and  design  optimization  framework  allowing  for  many  available 
analysis  solvers  or  custom  solvers  for  all  type  of  analysis  with  simulation  and  workflow  automation  [26]. 

There  are  many  options  available  to  us;  this  is  not  an  exhaustive  list  and  the  specific  modeling  language 
and  tool(s)  for  the  Vision  model  has  not  yet  been  decided.  The  "As  Is"  process  and  SETR  artifact 
dependencies  and  relationships  are  being  modeled  using  CORE,  as  discussed  in  Section  4.5.  The  CORE 
tool  was  created  before  SysML,  and  CORE  supports  a  number  of  commonly  used  SE  modeling  diagrams 
sequence,  activity,  N2,  and  Enhanced  Function  Flow  Block  Diagram,  and  increasingly  many  of  the 
diagrams  supported  in  SysML  tools.  Because  SysML  is  general,  there  are  possible  mappings  to  many 
types  of  modeling  languages  (as  is  true  for  UML  too)  [38]  as  well  as  support  for  programmatic 
interchange  based  on  the  XMI  standard.  This  may  rationalize  why  some  organizations  are  using  SysML 
as  an  integrating  framework,  that  is,  they  may  not  be  modeling  in  SysML,  but  they  are  using  SysML  (and 
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associated  tooling)  as  a  mapping  or  an  interchange  medium  between  different  modeling  languages  and 
environments.  While  the  SysML  and  UML  languages  and  tools  help  significantly  to  formalize  the 
expression,  exchange,  and  graphical  representation  of  system  models,  they  remain  ambiguous  and  in 
need  of  extensions  to  capture  the  specific  semantics  of  a  given  engineering  domain  [34]. 

Therefore,  our  team  will  use  modeling  notations  like  SysML  in  this  report.  However,  the  perspectives 
cited  in  this  section  reflect  on  why  the  Vision  must  go  beyond  and  use  other  more  semantically  rich  and 
precise  model  representations,  as  well  as  supporting  semantically  consistent  model  (digital)  interchange 
between  different  simulation  and  analysis  tools.  Our  efforts  planned  for  Phase  II  involve  a  modest 
demonstration,  and  we  fully  envision  that  we'll  need  to  use  other  modeling  approaches  that  support 
simulation,  analysis,  synthesis  and  generation. 


4.3  Straw  man 

During  the  kickoff  meeting,  it  was  decided  that  we  would  attempt  to  build  a  model  of  the  Vision. 
Therefore  following  good  modeling  guidelines,  we  started  with  a  context-level  representation  that  was 
derived  from  Integrated  Warfighter  Capability  (IWC)  graphic  associated  with  Task  3  shown  in  Figure  1. 
The  top  level  IWC  is  represented  using  a  SysML  Block  Definition  Diagram  (BDD)  diagram  as  shown  in 
Figure  11.  This  provided  a  way  to  reflect  that  the  effort  involves  characterizing  all  types  of  information 
that  is  necessary  to  design,  verify,  validate,  produce,  acquire  and  deploy  a  weapon  system.  We  used 
other  documents  describing  the  Operational  Concept  Document  of  Navy  Integration  and 
Interoperability  (l&l)  Environment  [29]  and  created  a  similar  diagram  as  shown  in  Figure  12.  Regardless 
of  the  content  and  modeling  approach  (SysML),  the  mere  existence  of  these  examples  stimulated 
significant  discussion  at  the  working  session  and  clarified  for  the  team  what  is  meant  by  modeling  the 
Vision  and  the  concept  of  capturing  all  information  in  ''system"  model. 


Figure  11.  SysML  Context  of  IWC  Vision 


Conceptually,  the  Vision  model  will  be  a  reference  architecture  (or  metamodel)  of  a  multi-level,  multi- 
domain  integrated  engineering  approach  to  support  the  IWC.  It  is  not  going  to  describe  a  specific 
instance  of  a  system;  instead  it  will  ideally  characterize  all  of  the  types  of  information  related  to  the 
design  including  characterizations  of  the  supporting  environmental  resources  for  simulation  and 
analyses,  design  parameters  and  constraints,  verification  and  flight  readiness  evidence,  and  associated 
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risk-based  signoffs.  Ultimately,  it  should  include  everything  to  the  Bill  of  Material  (BOM)  required  to 
manufacture  and  produce  the  system  (or  in  the  future  the  specifications  for  3D  printing). 

It  was  decided  to  scope  the  effort  at  a  Program  of  Record  (POR)  (e.g.,  F18  with  weapons,  CH-53). 
Referring  to  the  BDD  in  Figure  11  and  Figure  12,  a  POR  relates  to  the  Integrated  Capability  Technical 
Baseline  (ICTB)  block.  The  ICTB  block  is  also  represented  in  Figure  11  (the  Integrated  Warfighter 
Capability  BDD).  From  the  perspective  of  the  l&l  Environment  BDD,  relationships  from  the  ICTB  block  to 
the  Mission  Technical  Baseline  (essentially  where  the  requirements  for  the  ICTB  are  derived),  and 
System/Program  Technical  Baseline  blocks  are  reflected.  All  of  these  blocks  relate  to  the  l&l  Repository. 
The  l&l  Repository  is  part  of  the  Navy's  Integration  and  Interoperability  Integrated  Capability 
Framework  vision  that  includes  an  enterprise  data  environment  for  storing  and  sharing  DoDAF 
architecture  and  other  supporting  l&l  data  in  a  common  format  with  common  ontologies  to  support 
cross-correlation  and  alignment  [29].  These  BDDs  provide  two  perspectives  on  the  relationships  to  the 
ICTB  within  the  NAVAIR  vision,  but  this  is  still  at  a  very  high  level.  In  order  to  complete  a  representation 
of  the  Vision  it  will  be  necessary  to  formalize: 

■  All  information  as  typed  data  elements,  which  can  be  represented  as  attributes  in  a  model 

■  Data  flows  reflecting  the  data  dependencies  between  blocks 

o  BDD  diagrams  often  have  associated  Internal  Block  Diagrams  (IBD),  which  show 

hierarchically  lower-level  diagrams  with  the  corresponding  data  flow  between  the  lower- 
level  blocks 

o  As  another  type  of  example.  Figure  13  shows  that  the  Vision  must  not  only  be  able  to 

characterize  the  elements  of  the  vehicle  system,  but  should  also  characterize  the  elements 
within  the  overarching  environment  that  show  uses  or  dependencies  to  resources  such  as 
simulation,  test  environment,  instrumentation  and  logging 

o  Surrogates  would  also  be  represented  by  blocks 

■  Control  flow  reflecting  both  sequential  and  concurrent  flows 

o  Activity  diagrams  in  SysML  can  represent  both  control  flow,  and  the  associated  data  flows 
that  would  be  associated  with  flows  within  an  Internal  Block  Diagram  (IBD) 


There  are  other  behavioral  views  (e.g.,  sequence  and  state  diagrams)  and  constraint  views  (parametrics) 
that  would  be  necessary  to  fully  characterize  the  information  needed  to  produce  an  air  vehicle  system. 


Figure  12. 1  &  I  Environment 
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4.4  Acquisition  Guidance  Model  (AGM)  Model 

We  also  discussed  how  some  aspects  of  the  Vision  could  possibly  be  done  from  a  process  perspective 
using  other  modeling  techniques  such  as  Business  Process  Modeling  Notation  (BPMN).  This  is  when  the 
MITRE  Acquisition  Guidance  Model  (AGM)  model  effort  was  identified  as  a  possible  contributor  to  our 
effort. 

In  alignment  with  goals  for  Task  1,  we  will  continue  to  investigate,  and  leverage  existing  ideas  and 
advances  to  MBSE  through  discussions  with  industry  organizations.  MITRE  developed  the  AGM  [15].  The 
artifacts  and  model  representation  aligns  with  the  DoD  5000.02  phases,  and  embeds  guidance 
described  in  the  Defense  Acquisition  Guidebook  (DAG),  with  particular  emphasis  on  Chapter  4  of  the 
DAG,  and  the  relationships  between  SE  guidance  [18].  We  wanted  to  determine  if  we  could  leverage  it 
to  create  or  map  to  the  Vision  model.  At  worst  we  should  be  able  to  use  the  AGM  to  understand  the 
completeness  of  the  "As  Is"  process  as  we  develop  a  Vision.  This  would  help  people  understand  how  a 
process  would  work  when  transitioning  from  a  document-centric  operational  model  to  a  model-centric 
approach. 

The  AGM  was  developed  using  the  iGrafx®  tool  with  BPMN  [12].  The  AGM  provides  a  high-level 
characterization  of  the  activities,  events  and  messages  using  a  BPMN  notation  as  shown  in  Figure  14.  It 
provides  a  time-sequenced  perspective  on  the  process  using  swim  lanes  to  more  easily  understand  the 
different  functional  capabilities  or  responsibilities;  specifically  it  documents  activities,  messages,  and 
products.  The  AGM  provides  a  mechanism  to  examine  how  SE  fits  into  of  range  of  actions  within  an 
acquisition  program,  how  SE  leverages  results  of  other  acquisition  activities  and  which  SE  products 
support  acquisition  decisions.  Some  SMEs  from  organizations  such  as  Defense  Acquisition  University 
(DAU)  instructors,  as  well  as  program  managers  from  Acquisition  Category  (ACAT)  1  programs  have 
reviewed  the  AGM.  While  this  could  be  useful  as  a  completeness  check,  it  is  general  and  the  generic 
artifacts  don't  map  very  granularly  to  the  actual  artifacts  that  NAVAIR  must  produce  from  both  a 
functional  and  airworthiness,  safety,  and  program  execution  (see  Section  4.5). 


®  We  make  no  claims  about  this  particular  modeling  tool.  It  is  a  tools  used  by  MITRE. 
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Figure  4:  Meesagefi  &  Message  Flows 


Figure  14.  AGM  Representations 


4.5  "As  Is"  Artifact  Analysis 

A  key  guideline  for  the  SE  process  is  the  SETR  process.  As  reflected  in  Figure  15,  the  SETR  handbook  is 
one  of  the  guiding  instruments  for  the  SETR  process.  It  characterizes  the  Roles  of  the  individuals  that 
must  perform  Activities  to  develop  the  Artifacts  that  result  in  Measures  to  make  the  decisions. 

An  approach  that  we  used  in  the  past  when  assisting  organizations  in  adopting  MBSE  is  to  examine  all  of 
the  required  artifacts  that  are  produced  in  their  current  processes,  and  them  examine  the  ways  models 
and  the  associated  model-based  automation  (e.g.,  simulation,  analysis,  synthesis,  generation,  test 
generation,  etc.)  can  be  applied  to  those  models  to  reduce  or  eliminate  "paper-based"  documentation 
and/or  modify  or  eliminate  the  manual  processes  [8].  We  have  heard  similar  stories  from  some  of  the 
industry  organizations  in  our  preliminary  discussions  about  using  a  combined  bottom-up  and  top-down 
approach  to  transform  their  MBSE  efforts,  and  are  expecting  to  be  briefed  on  details  once  we  have  the 
appropriate  confidentiality/proprietary  agreements  in  place. 
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PEO 


Figure  15.  System  Engineering  Technical  Review  Process  Handbook 

Our  NAVAIR  team  categorized  about  330  artifacts,  which  has  been  reduced  to  about  140  that  are 
produced  as  part  of  the  SETR  process,  as  shown  in  Table  3.  The  list  of  artifacts  was  originally  captured  in 
a  categorized  spreadsheet,  however  this  representation  has  some  limitations,  as  it  was  difficult  to 
characterize  artifact  dependencies  and  relationship,  as  well  as  the  timing  in  the  process  where  they  are 
created  or  used  in  reviews.  Therefore  our  team  partners  developed  a  CORE  model  using  a  data-driven 
approach  to  characterize  a  subset  of  some  of  the  artifacts  (captured  in  spreadsheet).  The  concept 
demonstrates  how  the  data-driven  approach  allows  the  inherent  process  to  be  derived  from  the  artifact 
relationships  and  data  dependencies.  The  CORE  tool  has  different  types  of  graphical  views  that  render 
different  type  of  relationships  (e.g.,  data  flow,  data  dependency)  automatically  from  information 
collected  in  a  data-driven  way.  Two  different  examples  are  shown  in  Figure  16  and  Figure  17.  NOTE:  this 
is  not  the  NAVAIR  process,  rather  this  is  just  a  derived  process  that  relates  to  the  inherent  artifact 
relationships.  As  reflected  in  Figure  10,  they  are  also  overlaying  a  process  based  on  International 
Organization  for  Standardization  (ISO)  15288,  also  with  a  mapping  back  to  DoD  5000.02.  ISO-15288, 
published  by  ISO,  is  a  world-wide  standard  for  systems  and  software  engineering  lifecycle  processes. 
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Table  3.  "As  Is"  Artifacts 


SE 

Partici 

pation 

Activities 

Product 

Tools  used 

Model 

Types 

Analysis 

Simulation 

Generation 

Test 

Generation 

High 

1-  Design  (Software) 

Software  Design  Description 

High 

1-  Design  (Software) 

Software  Interface  Design  Description  (SIDD) 

High 

1-  Design  (Software) 

Software  Product  Baseline 

High 

1-  Design  (Software) 

Software  Build  Plan 

High 

1-  Design  (Software) 

Software  Architecture  Description  (SAD) 

High 

1-  Design  (Software) 

Software  Code  Documentation 

High 

1-  Design  (Software) 

Software  Measures 

High 

1-  Design  (System) 

Air  Vehicle  Specification 

DOORS 

High 

1-  Design  (System) 

Engineering  Drawings 

varies 

Simulink, 

Control 

Sim  and 

High 

1-  Design  (System) 

Flight  Control  Detailed  Design  Report 

Stateflow 

Laws 

Satisfiability 

Matlab 

Code 

Code 

High 

1-  Design  (System) 

Integrated  Architecture  Model  - 1  Requirements  Analysis  Viewpoints 

DoDAF 

High 

1-  Design  (System) 

Integrated  Architecture  Model  -  2  Functional  Analysis  Viewpoints  (OV-3, 

DoDAF 

High 

1-  Design  (System) 

Integrated  Architecture  Model  -  3  Allocation  Baseline 

DoDAF 

High 

1-  Design  (System) 

Integrated  Architecture  Model  (lAM)  (DoDAF) 

DoDAF 

High 

1-  Design  (System) 

Modeling  and  Simulation  Report 

Varies 

High 

1-  Design  (System) 

Sub-System  Design  Documentation 

High 

1-  Design  (System) 

System  Design  Documentation  (SDD) 

High 

1-  Design  (System) 

System  Design  Specification  (SDS) 

DOORS 

High 

1-  Design  (System) 

System  Specification  (SS) 

DOORS 

High 

1-  Design  (System) 

Sub-System  Design  Description  &  Analysis  Report 

High 

1-  Design  (System) 

Sub-System  Specification  Documentation 

High 

1-  Design  (System) 

Training  System  Functional  Description  (TSFD) 

High 

1-  Requirements 

Concept  of  Operations  (CONORS) 

High 

1-  Requirements 

Design  Reference  Mission  Profile 

High 

1-  Requirements  (System) 

Software  Requirements  Description  (SRD) 

DOORS 

High 

1-  Requirements  (System) 

Software  Requirements  Specification  (SRS) 

DOORS 

High 

1-  Requirements  (System) 

Software  Requirements  Traceability  Matrix 

High 

1-  Requirements  (System) 

Net-Ready  Key  Performance  Parameters  (NR-KPP) 

RSA  (?) 

High 

1-  Requirements  (System) 

Requirements  Traceability  Matrix/Product  (RTM) 

High 

1-  Requirements  (System) 

Requirements  Verification  Traceability  Matrix  (RVTM) 

High 

1-  Requirements  (System) 

Capabilities  Development  Document  (CDD) 

High 

1-  Requirements  (System) 

Capabilities  Production  Document  (CPD) 

High 

1-  Requirements  (System) 

Contractor  System  Specification  (SS) 

High 

1-Design  (System  Safety) 

System  Hazard  Analysis 

High 

1-Design  (System  Safety) 

Critical  Safety  Item  (CSI) 

High 

1-Design  (System  Safety) 

Critical  Safety  ltem(CSI)/Critical  Application  Item  (CAI)  Management  Plan 

High 

1-SE  Mgmt 

System  Engineering  Management  Plan  (SEMP) 

High 

1-SE  Mgmt 

System  Engineering  Plan  (SEP) 

All  Elements 


0.0  Acquire  System 

1.0  Develop  System  Concepts 

1 . 1  Define  Problem 

1.2  Define  System/Program  Goals 

1.3  Define  Development  Objectives 

1 .4  Define  System/Program  Assumptions 

1.5  Define  System/Program  Constraints 

1.6  Conduct  Program  Trade-off  Analysis 

1.7  Select  Initial  Capabilities 


2.1  Define  User  Requirements 

2.1.1  Define  User  Problem 

2.1.2  Analyze  User  Mission 

2.1.3  Analyze  User  Needs 

2.1.4  Define  External  Scope  and  Bounds 

2.1.5  Analyze  User  Value  Design 

2.2  Define  System  Requirements 

2.2.1  Define  System  Problem 

2.2.2  Define  System  Operations 

2.2.3  Analyze  System  Requirements 

2.2.4  Define  System  Scope  and  Bounds 

2.2.5  Analyze  System  Value  Design 
3.0  Develop  Architecture 

3.1  Define  Functions 

3.1.1  Decompose  Functions 

3.1.2  Allocate  Functions 

3.2  Define  Behaviors 

3.3  Design  Functional  Architecture 

3.4  Define  Operational  Architecture 

3.5  Define  Physical  Architecture 


a  □  >  (i*  l  r-- 1^-  j-'i.  "  Kb  O  | 

3  n 

Figure  16.  Derived  Process 
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Figure  17.  "As  Is"  Artifacts  and  Process  Relationships 

We  know  from  discussions  with  organizations,  and  at  the  meetings  with  MITRE,  that  organizations  often 
use  a  combination  of  top-down  and  bottom-up  as  they  evolve  into  a  model-driven  approach. 
Organizations  tend  to  roll  out  the  adoption  of  modeling  in  a  more  evolutionary  manner.  While  this  is 
interesting,  the  research  goal  is  not  evolution  rather  it  is  to  assess  the  technical  feasibility  to  radically 
transform  SE  through  MBSE.  However,  context  does  matter,  because  we  also  need  to  assess  the 
technical  feasibility  in  terms  of  the  types  of  systems  and  airworthiness  needs  required  by  NAVAIR;  part 
of  this  process  is  exposing  areas  where  there  is  expert  judgment,  historical  evidence  and  rules  of 
thumbs  that  are  used  in  the  decision-making  process,  which  needs  to  be  better  formalized  and 
embedded  in  knowledge  within  the  Vision. 


4.5.1  Model  Transformation  Rather  Than  Model  Evolution 

To  reflect  on  the  concept  of  model  transformation  rather  than  model  evolution,  we  provide  the 
following  example  to  describe  how  model-based  automation  can  completely  eliminate  manual  effort 
and  result  in  radical  transformation  of  the  ''As  Is"  process  through  an  automated  workflow.  The 
following  provides  a  scenario  for  how  to  think  about  using  models  to  replace  artifacts,  and  more 
importantly  how  model-based  automation  subsumes  manual  efforts.  Referring  to  the  artifact  "Flight 
Control  Detailed  Design  Report,"  which  is  highlighted  in  Table  3,  this  type  of  artifact  would: 

■  Represent  "Control  Law"  in  a  model 

o  Simulink^  and  Stateflow  are  commonly  used  to  model  control  laws  (e.g.,  F16,  F18,  F35) 

■  Automated  analysis  that  exists  today,  (e.g.,  it  has  been  applied  to  F35)  would  include: 
o  Satisfiability:  proving  that  each  thread  through  the  model  has  no  contradictions 

(mathematical  consistency) 

■  Simulation 

o  Simulation  of  Simulink  models  is  often  done  using  Matlab 
o  Support  high-fidelity  simulation  using  Matlab 

o  Support  higher  fidelity  with  real-time  execution  within  the  surrogate  or  prototype  system 
implementation  or  actual  hardware  though  automatic  code  generation 

■  Synthesis  or  generation 


We  are  not  promoting  Simulink,  we  use  it  as  an  example,  because  it  is  almost  a  defacto  standard  for  control 
system  modeling  and  simulation 
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o  Code  generation  from  Simulink  models  can  be  provided  by  Mathworks  and  other 
commercial  products 

o  Automatic  test  generation  directly  from  Simulink  models 
o  Automatic  test  driver  generation 

■  The  test  vectors  are  transformed  into  various  types  of  test  drivers  that  are  run  both  against  a 
Matlab  simulation  and  the  auto-generated  code;  if  all  tests  pass  (the  actual  output  equals  the 
expected  output)  in  both  the  simulation  and  generated  code  execution  environments  then  there 
is  a  strong  verification  argument  that  the  code  satisfies  the  specification 

o  Organizations  run  the  test  through  both  the  simulation  and  code,  because  organizations 
have  been  able  to  find  errors  in  the  code  generation  (Risk  reduction  argument  for  using 
model-based  tools) 

■  Code  coverage  tools  such  as  LDRA  and  VectorCast  have  been  used  to  show  that  the  tests 
provide  Modified  Condition/Decision  (MC/DC)  coverage 

o  Code  coverage  measurement,  which  provides  quantified  risk  reduction  evidence 

■  The  Mathworks  code  generation  uses  a  particular  algorithm  that  produces  code  that  is 
"deadlock"  free 

o  Eliminates  concurrency  analysis 

These  are  types  of  model-based  automation  that  leverage  models  to  "Cross  the  Virtual  V."  While  this 
can  be,  and  is  commonly  done  on  low-level  high-fidelity  models,  we  are  also  interested  in  applying  this 
type  of  concept  at  the  upper-levels  of  the  "V"  with  varying  levels  of  fidelity  that  provide  integration  of 
model  and  model  automation  at  different  levels  of  the  "V"  as  reflected  Figure  18.  This  figure  is  not 
complete,  as  there  are  many  lines  not  included  between  all  of  the  different  types  of  model,  but  this 
diagram  has  been  useful  in  conversations  about  "Crossing  the  Virtual  V"  during  working  sessions.  These 
types  of  models  mentioned  at  the  upper-level  of  this  V  may  not  directly  relate  to  the  "As  Is"  artifacts, 
but  they  must  be  defined  in  the  Vision. 
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Verification  and  Validation 
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Figure  18.  Notional  Multi-level  Crossing  the  V 


We  have  continually  discussed  the  notion  of  "Crossing  the  Virtual  V"  as  an  important  way  to  assess 
system  design  concepts  in  the  context  of  mission  scenarios.  However,  in  discussions  with  organizations, 
there  are  some  that  believe  that  the  notion  of  a  "V"  is  a  historic  manifestation  of  "the"  traditional  flow 
of  document-driven  work,  and  we  should  eliminate  the  use  of  the  "V"  as  part  of  systems  engineering 
dogma  as  it  is  counterproductive  to  embracing  approaches  that  support  the  continuous  integration  of 
digital  artifacts.  The  "V"  introduces  points  for  disconnects  and  failure.  What's  more  critical  in  the  Vision 
is  continuous  integration  of  various  types  of  digital  assets,  with  varying  levels  of  abstraction  and  various 
degrees  of  fidelity.  Section  4.5.2  provides  some  discussion  on  this  point  using  examples  to  further  clarify 
the  notion  of  physical  surrogates,  and  support  the  argument  that  the  "V"  may  not  be  a  good  metaphor. 


4.5.2  Digital  and  Physical  Surrogates 

The  concept  of  "model  everything"  relies  heavily  on  digital  assets  such  as  physical  surrogates,  existing 
system  or  component  re-purposed  for  new  concept  exploration  and  prototyping.  Our  NAVAIR  team 
created  a  concept  for  representing  System  Level  Maturity,  as  shown  in  Figure  19.  It  reflects  on  the  idea 
that  as  we  are  attempting  to  "Cross  the  Virtual  V"  and  will  rely  on  physical  surrogates,  which  is 
commonly  done  today,  both  in  aerospace  and  other  domains  such  as  auto  racing.  The  actual  airframe, 
shown  on  the  right  side  of  the  V  matures  (top-to-bottom)  and  the  actual  aircraft  is  first  flow  (e.g.,  F-35, 
15-December-2006)  long  before  many  of  the  complex  software  intensive  systems  are  developed  and 
integrated,  as  the  aircraft  airframe  and  new  materials  are  being  evaluated.  Key  early  capabilities  such  as 
software  for  the  control  laws  to  fly  the  aircraft  are  often  evolved  from  earlier  aircraft  systems  (e.g., 
many  versions  of  MATRIXx  and/or  Simulink  models  have  been  evolved  for  years,  and  will  continue  to  be 
evolved  for  years).  Yet,  all  of  these  systems  are  continually  refined  and  as  the  timeline  of  system 
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capabilities  mature,  new  capabilities  are  added  to  the  system.  As  the  subtitle  "integrate  and  build" 
suggests,  continuous  integration  is  the  key  to  this  paradigm.  Formalized  interfaces  are  required  for 
integration,  and  the  semantics  for  the  interfaces  often  need  to  be  formalized:  1)  structurally,  2) 
behaviorally,  and  3)  temporally,  in  order  to  use  surrogates  and  simulations.  Document-based 
specifications  do  not  formalize  these,  however  some  modeling  approaches  can,  and  with  semantic 
formalization,  automated  verification  can  be  supported  directly  from  the  models. 


System  Level  Maturity  Development  and  Verification** 


a* 


integrate  then  build"  * 

System  consists  of  "design"  (payload,  avionics,  hardware,  etc.) 
and  the  "truck"  to  carry  the  "design" 

Design/Payload  Maturity: 


Modeling 

O  S 

High  level  need:  Aircraft 
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*  System  Architecture  Virtual  Integration:  An  Industrial  Case  CMU/SEI-2009-TR-017 
** Ernest  S.  "Turk”  Tavares,  Jr.  and  Larry  Smith _ 


Figure  19.  System  Level  Maturity  Representation  for  Development  and  Verification 


The  key  issue  with  the  traditional  V  as  shown  in  Figure  19  is  that  the  V  model  does  imply  time 
progressing  chronologically  from  left-to-right,  and  while  the  concept  of  "Crossing  the  Virtual  V"  is 
notionally  easy  to  understand,  and  as  pointed  out  by  one  of  our  discussion  groups,  the  V  model  does  not 
work  well  in  a  model-based  paradigm.  Figure  20  presents  the  same  logical  progression  of  the  design  and 
verification  that  is  aligned  with  the  traditional  lifecycle  phases,  and  progress  does  correspond  with  a  left- 
to-right  time  ordering. 
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Figure  20.  Non-V  Model  System  Level  Maturity  Representation  for  Development  and  Verification 


4.5.3  Vision  Model  Reference  Architecture 

Generic  processes  and  guidelines  for  characterizing  requirements,  designs,  variants,  verification 
evidence  and  risks  using  a  document-based  approach  simplifies  the  way  process  artifacts  are  titled  or 
labeled  as  reflected  in  Table  3.  However,  as  shown  in  Figure  21,  there  are  many  types  of  major  systems 
elements,  and  subsystems  within  an  aircraft  system.  Therefore,  we  can  image  that  the  model  of  the 
Vision  must  need  to  use  a  type  of  reference  architecture  in  the  characterization  of  an  integrated  set  of 
model  types  (software,  electrical,  hydraulic,  human  interface,  etc.)  that  represent  all  of  the  engineering 
aspects  of  an  aircraft  system  (i.e.,  the  ''total"  system  model).  There  must  also  be  ways  to  characterize 
different  types  of  elements,  for  example,  a  winged  aircraft  may  not  have  rotors,  and  a  UAV  may  not 
have  avionics  displays. 
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Figure  21.  Model  Topology  Often  Mirrors  Architecture  of  System 


4.6  Scope  to  Program  of  Record  through  Digital  Critical  Design  Review 

The  scope  reduced  at  the  last  working  session  was  to  focus  on  the  process  up  to  critical  design  review 
(CDR),  for  the  ''As  Is/'  and  for  the  Vision  model  and  risk  framework  for  a  POR.  It  was  thought  that  the 
technical  reviews  are  good  "checkpoints"  since  they  focus  on  different  decisions  and  levels  of 
engineering  content  that  would  need  to  be  represented  in  the  models.  Only  the  PDR  and  CDR  are 
always  required.  Other  reviews  ASR,  SRR,  SFR,  TRR,  SVR,  PRR  may  or  may  not  be  required  on  a  given 
program.  Ideally,  we're  looking  for  a  new  concept:  Digital  design  from  CDR  artifacts  (DCDR).  We  want 
to  investigate  a  more  continuous  notion  of  PDR  and  CDR  (or  DCRD)  where  reviews  of  models  and 
analysis  artifacts  occur  everyday  until  all  of  the  required  evidence  is  provided  in  order  to  support 
contracts  and  signoffs. 


4.6.1  Context  for  Program  of  Record  System 

Along  with  discussions  with  organizations  external  to  NAVAIR,  we  have  had  preliminary  discussions  and 
are  also  planning  discussions  and  demonstrations  to  better  understand  the  context  of  the  current 
environment  in  order  to  more  precisely  characterize  the  information  that  is  formalized  in  the  form  of 
models,  simulation,  and  analytical  data.  The  context  for  the  POR  starts  from  environmental  aspects  at 
the  mission-level.  For  many  efforts  organizations  often  start  with  an  OV-1  diagram  of  the  mission-level 
with  systems-of-system  (SoS)  level  interactions.  The  operational  views  decompose  the  mission  within 
context  of  the  situation,  and  provide  different  viewpoints  that  describe  the  tasks  and  activities 
operational  elements,  and  resource  flow  exchanges  required  to  conduct  operations. 

Building  on  lessons  learned  creating  early  DoDAF  models,  analyses  have  uncovered  that  interoperating 
at  the  lowest  (data)  levels  is  insufficient  for  scenarios,  and  scenarios  require  behaviors;  missing  at  the 
data  level.  DoDAF  does  not  accommodate  other  scenario  requirements  (e.g.,  conditions  assumptions) 
very  well,  and  is  insufficient  to  fully  characterize  the  dynamic  needed  for  analysis.  This  is  not  surprising. 
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as  DoDAF  has  mappings  to  SysML  [38],  and  we  have  discussed  some  of  the  limitation  of  SysML  in 
Section  4.2,  especially  related  to  dynamics  of  models. 


As  a  result,  study  views  were  created  to  address  a  number  of  challenges  at  this  level.  A  mission-level 
SoS  analysis  begins  with  formalization  through  Study  Views,  as  reflected  in  Figure  22.  Study  views 
provide  structure  and  a  common  context  that  acts  as  a  basis  for  framing  and  bounding  the  functional 
decomposition  of  DoDAF  products.  Study  views  formalize  the  need  and  intent,  provide  a  situational 
context  and  influencing  factors  to  frame  and  bound  the  functions  and  activities  of  the  mission  and 
scenarios  that  ultimately  lead  into  corresponding  representations  of  the  Mission  and  System 
Capabilities  (i.e.,  the  capabilities  for  the  FOR).  These  capability  representations  are  further  analyzed 
using  modeling  and  simulation  and  corresponding  analysis  capabilities.  The  outputs  of  which  are  then 
formalized  in  terms  of  DoDAF  artifacts.  This  information  will  form  the  analysis  boundaries  for  the 
System  Capabilities  information  that  is  to  be  captured  in  the  Vision  model. 


bdd  [Package]  Environment  [  Environmental  Context 


Figure  22.  Mission  Context  for  System  Capability^ 


4.6.2  Case  Studies  and  Existence  Cases 

We  have  bounded  the  scope  of  this  effort.  Figure  22  identifies  the  context  level  view  for  the  definition 
of  a  POR  system  capability.  It  does  it  in  the  context  of  a  Study  View,  which  is  a  more  abstract  view  than 
a  DoDAF  or  SysML  characterization.  The  ''As  Is"  artifacts  discussed  in  Section  4.5,  constrained  in  time  to 
the  CDR  limits  the  scope  moving  to  the  right  as  reflected  by  Figure  21. 


^  Image  source:  Thomas  Thompson,  Enabling  Architecture  Interoperability  Initiative,  B210-001D-0051  Unclassified. 
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We  are  still  in  search  of  a  case  study.  It  is  desirable  to  have  a  case  study  that  can  provide  technical 
design,  cost,  airworthiness  and  risk  data.  NAVAIR  identified  in  the  final  week  of  this  project  a  possible 
candidate,  but  if  that  information  cannot  be  provided,  we  will  look  to  develop  our  own  case  study 
example  or  transform  some  existence  cases  [22]  into  scenarios. 

One  possible  candidate  was  developed  as  part  of  a  Simulation  Based  Acquisition  experiment,  where  a 
low-fidelity  abstract  model  of  an  F-35  slow  takeoff  vertical  landing  (STOVL)  state  machine  model  was 
integrated  with  high-fidelity  control  laws  model  [7].  This  was  a  virtual  representation  of  system 
capabilities,  where  two  different  models  were  used  to  automatically  generate  simulation  code.  The 
models  were  analyzed  using  a  formal  methods  tool  (i.e.,  theorem  prover),  and  test  vectors  were 
automatically  generated  to  illustrate  both  the  integration  of  low  and  high  fidelity  models,  synthesis, 
simulation,  and  model-based  verification.  No  airworthiness  requirements  were  applied  to  this  example. 

Another  possible  candidate  was  demonstrated  by  a  commercial  MBSE  tool  company  that  we'll  be 
meeting  with  as  part  of  our  Task  1  discussions.  This  model  illustrates  a  UAV  capability  to  support  the 
mission  of  aerial  artifact  camera-based  examination  (e.g.,  water-based  wind  turbines),  and 
demonstrates  aspects  such  as:  flight  control,  flight  dynamic,  propulsion  and  energy,  and  mission 
execution.  This  model  may  be  supplied  under  our  confidentiality  agreement.  This  model  does  not 
address  airworthiness,  but  could  be  tailored  to  support  this  type  of  required  analysis. 
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5  Task  4  -  Integrated  Framework  for  Risk  Identification  and  Management 


We  want  to  investigate  the  use  of  risk-based  models  that  align  with  the  Vision  to  ''model  everything/' 
This  involves  how  the  Vision  should  include  integrated  risk  identification  and  management  as  reflected 
in  Figure  10.  While  there  are  many  classes  of  risks  to  manage,  for  NAVAIR  there  are  fundamentally  two 
key  classes  of  risk  that  must  be  considered: 

■  Airworthiness  and  Safety  (most  critical  in  Technical  Feasibility  assessment) 

■  Program  execution  (cost,  schedule  and  performance) 

There  are  also  two  complementary  views  of  model-based  acquisition  with  respect  to  risk: 

■  Risks  introduced  by  modeling  deficiencies  and  risks  reduced  by  enhanced  use  of  modeling 

■  Modeling  to  predict  or  assess  risks  (i.e.,  modeling  for  uncertainty  quantification  in  acquisition 
and  in  the  use  of  models) 


5.1  Predictive  Models  for  Risk 

The  SERC  team  has  used  models  in  the  prediction  of  risk,  and  plans  to  use  predictive  analytic  models  to 
support  risk  identification  and  management.  More  generally  we  can  use  models  to  provide  risk 
quantification  for  almost  all  types  of  decisions  that  are  made  by  stakeholders  (e.g.,  model-based 
reviews).  As  an  example,  we  created  a  Bayesian  model  using  factors  derived  from  the  Airworthiness 
standard  MIL-HDBK-516B  [20]  as  shown  in  Figure  23.  This  is  conceptually  similar  to  the  approach  we  are 
using  on  an  FAA  NextGen  research  task  for  collaborative  risk-informed  decision-making  [4][5][6].  The 
key  characteristics  of  the  approach  are  they  ensure  that  all  factors  are  considered  in  the  decision¬ 
making  process,  and  that  all  classes  of  stakeholders  are  adequately  represented  in  the  decision  making 
process.  A  systematic  and  comprehensive  treatment  of  all  relevant  factors  provides  better  risk 
identification. 

We  used  this  model  and  an  example  from  a  true  story  related  to  a  C130  Weapon  Delivery  system  to 
illustrate  the  concept.  While  this  model  is  notional  at  this  time,  this  example  started  a  discussion  with 
the  team  about  how  stochastic  (probabilistic)  models  can  play  an  important  part  of  the  Vision  as  they 
formalize  many  aspects  of  the  human  decision  making  process  that  will  be  important  at  many  gates, 
reviews,  and  decision  points  of  the  Vision  concept.  Each  factor  covers  a  specific  aspect  of  airworthiness, 
to  ensure  that  all  possible  uncertainties  and  risk  are  considered  in  the  quantification  of  risk.  The  risk 
index  is  a  probability  distribution,  where  for  example,  the  mean  can  map  to  quantities  in  a  risk  matrix. 
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5.2  Risk  Framework  Approach  to  Uncertainty  Quantification  Modeling  and  Prediction 

The  SERC  team  has  also  been  working  with  NAVSEA  to  develop  a  framework  and  approach  to 
Uncertainty  Quantification  modeling  and  prediction.  The  approach  has  three  main  components: 

■  Identifying  the  design,  test  and  modeling  factors  at  different  system  scales 

■  Analyzing  the  uncertainty,  variability,  and  error  in  design  implementation,  testing,  and  modeling 

■  Using  experimental  design  methods  to  assess  the  contributions  and  interactions  to  system 
(airworthiness  and  safety)  and  program  execution  risks 

The  risk  modeling  and  analysis  approach  also  addresses  potential  errors  and  uncertainties  in  the 
overuse  of  limited  data.  Ideally: 

■  One  data  set  is  used  to  identify  critical  factors 

■  A  second  independent  data  set  is  used  to  develop  the  models 

■  A  third  independent  data  set  is  used  to  calibrate  the  models 

■  A  fourth  independent  data  set  is  used  to  assess  the  expected  error  in  model  results 

In  practice  data  sets,  surrogate  vehicle  test  data,  etc.  are  limited.  Bootstrap  methods  use  repeated 
resampling  of  the  data  and  repeating  the  modeling  and  analysis  process  to  obtain  a  statistical  estimate 
of  the  uncertainty  in  model-based  acquisition  given  the  available  data.  Further  analysis  reveals  the 
value  -  reduction  in  uncertainty  -  for  additional  data. 

These  types  of  models  capture  and  embed  knowledge  associated  with  expert  judgment,  historical 
evidence  and  rules  of  thumbs  that  are  used  in  the  decision-making  process. 
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5.3  Scope  of  the  Risk  Framework 

We  worked  with  our  NAVAIR  team  members  to  determine  the  scope  for  the  risk  framework.  Key  to  the 
representation  of  the  models  (and  Task  3)  to  support  risk  identification  and  management  is  to 
characterize  the  types  of  evidence  that  are  required  for  Flight  clearance  and  Flight  readiness.  It  is 
important  to  understand  how  the  models  are  developed  and  derived  in  order  to  understand  the  risk 
strategies  that  must  be  in  place  for  assessing  the  evidence  for  flight  clearance. 

The  process  for  risk  under  consideration  for  this  SE  transformation  covers  system  development  from 
Milestone  A  to  CDR  (at  least  for  now).  These  questions  related  to  risk  also  helped  to  refine  the  scope  for 
Task  3,  and  introduced  a  new  term  Digital  CDR  (DCDR),  with  a  heavy  emphasis  on  digitally-derived 
evidence  for  airworthiness  and  safety,  but  to  also  include  program  execution. 

In  both  preliminary  discussions  with  organizations  and  our  NAVAIR  team,  it  is  recognized  that  it  is 
important  to  quantify  "margins"  and  "sensitivities"  as  a  way  to  quantify  risk. 

As  an  example,  one  of  the  organizations  (in  our  preliminary  Task  1  discussion)  creates  new  types  of 
advanced  material  for  a  system.  They  cited  a  particular  effort  working  with  advances  in  new  material 
and  processes  at  the  nanoscale.  At  the  component  level  the  margins  seemed  acceptable.  However 
after  composing  the  components,  margins  propagated  to  unacceptable  levels  in  the  final  integrated 
form. 

Risk  implies  probabilities  of  what  might  go  wrong  or  might  not  happen  (on  time  or  due  to  the  degree 
expected),  and  some  distribution  of  magnitude  of  consequences.  This  requires  "impossible  certainty"  of 
the  degree  of  uncertainty  and  advance  knowledge  of  the  likelihood  and  effects  of  unidentified  events 
and  factors.  Therefore,  we  suggested  that  a  better  framework  might  be  to  work  in  terms  of  design 
margin.  Design  margin  is  more  closely  related  to  design.  Design  margin  is  how  much  room  there  is  for  a 
subsystem  or  component  to  perform  less  well  than  expected  or  to  have  greater  burdens  than  expected 
until  it  becomes  a  problem.  In  some  cases,  e.g.  weight,  any  increase  adds  to  total  weight,  so  instead  of  a 
weight  margin,  we  might  want  to  think  in  terms  of  sensitivities  (sensitivity  in  increase  in  total  weight, 
time,  cost,  etc.  to  a  percentage  increase  in  the  component  weight,  time,  power  draw,  etc.).  This  creates 
a  number  of  questions  for  this  task,  for  example: 

■  Can  we  use  models  to  see  how  much  design  margin  there  is  in  a  system  -  specifically  when  we 
cannot  push  the  system  to  failure;  which  types  of  models  and  how  can  we  use  them  to  estimate 
the  conditions  under  which  the  system  begins  to  exhibit  unstable  response 
o  In  control  systems  analysis  this  is  often  taken  to  be  the  3dB  point  -  the  frequency  of  input 
variation  at  which  the  output-to-input  ratio  is  half  what  it  was  for  low  frequency  change,  or 
the  90-degree  phase-shift  point,  where  the  frequency  of  input  variation  at  which  the  system 
response  lags  by  90  degrees 

o  Control  systems  analysis  methods  also  address  the  acceleration,  velocity  and  displacement 
limits  at  which  the  system  dynamics  change 

o  Failures  are  often  associated  with  transitions  from  linear  to  highly  non-linear  regimes;  often 
the  structure,  interactions  and/or  dynamics  change  in  these  regions  (e.g.,  insulators  or 
isolators  fail,  etc.)  -  e.g.,  the  acceleration,  velocity  and  displacement  limits  at  which  the 
system  transitions  from  linear  to  non-linear  response 
o  Models  that  are  relevant  in  the  "linear"  regime  will  give  erroneous  results  in  the  non-linear 
regime 

o  Models  that  do  not  represent  the  dynamics  that  change  the  structure  of  a  system  (e.g., 
insulation  wearing  off  causing  a  short-circuit,  structural  failure  of  a  linkage,  strain  transitions 
from  elastic  to  plastic  deformation,  etc.)  will  give  erroneous  results 
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Mechanical  or  electro-mechanical  control  and  isolation  systems  are  good  examples,  and  important  for 
airworthiness.  Control  systems  work  within  a  limited  range.  Standard  control  system  analysis  examines 
the  frequency  response  and  looks  for  the  3dB  frequency,  i.e.  the  frequency  at  which  the  transfer 
function  is  half  of  the  low-frequency  value  (the  transfer  function  is  just  the  ratio  of  output-to-input). 
Other  limits  include  maximum  displacement,  velocity  and  acceleration  -  when  the  system  hits  hard- 
stops,  current  limits  etc. 

Surrogates  can  be  driven  with  increasing  frequency  inputs  to  find  the  3dB  point  without  having  to 
experience  the  failure.  The  input  parameters  of  virtual  models  are  often  "tuned"  to  match  the  3dB  point 
of  test  data,  and  then  used  to  extrapolate  to  find  the  3dB  point  of  hypothetical  systems.  Physically 
realistic  models  can  be  used  to  estimate  the  limiting  thresholds  of  stable  response,  provided  the  models 
and  inputs  are  adequately  calibrated  and  validated.  Special  consideration  is  needed  for  basic  physical 
processes  with  non-linear  response  in  the  regime  of  operation,  e.g.,  friction  between  moving  parts 
versus  friction  between  stationary  parts. 

Nested  control  loop  models  have  been  used  effectively  in  system  safety  modeling  and  analysis  [27]. 
The  outer  control  loops  detect  changes  in  the  response  behavior  of  inner  control  loops,  and  then  adjust 
the  parameters  of  the  inner  control  loops  to  bring  the  inner  loops  back  into  the  stable  regime.  The 
SERC  team  has  used  this  framework  to  model  driving  behavior  and  system  response  at  the  "safety 
limits." 

In  the  use  of  modeling  and  simulation,  there  are  different  types  of  simulation  with  different  levels  of 
fidelity.  A  significant  challenge  is  that  tools  don't  often  map  well  to  different  levels  of  abstractions. 
These  are  areas  to  frame  risk.  There  are  increasing  uses  of  model  transformation  from  one  level  or  to 
different  disciplines.  Model  transformation  and  model  consistency  between  these  views  becomes  a  risk 
issue. 

A  companion  concept  is  credibility  of  the  estimates  of  performance,  cost,  etc.  High  credibility  if  it  has 
worked  in  a  surrogate  system,  less  if  it  is  similar  to  something  demonstrated  in  a  surrogate  and  model 
extrapolation.  It  will  be  important  to  better  understand  model  extrapolations. 

■  Less  credibility  the  farther  the  model  extrapolation  is  extended 

■  Less  credibility  going  from  surrogate  system  to  bench  testing,  etc. 

■  Use  of  multi-scale  calibration  and  validation 

■  Use  of  progressive  model-based  design  confirmation  in  technical  reviews 
o  Subsystems  mature  and  are  integrated  at  different  rates 

o  Sometimes  early  decisions  are  needed  for  long-lead  time  items  whose  specifications  can  be 
confirmed  before  other  aspects  of  the  system  (e.g.,  final  control  system  parameter  values) 


5.4  Risk  Framework  Captures  Knowledge 

These  types  of  risk  frameworks  are  actually  knowledge  models  of  credibility  (not  models  of 
performance,  but  models  of  uncertainty).  Part  of  the  effort  on  modeling  the  "As  Is"  process  (Task  3)  is 
to  identify  and  then  formalize  within  the  models  the  information  and  associated  knowledge  for 
evidence-based  decisions  and  evidence-based  timing  of  decisions.  Other  considerations  and 
opportunities: 

■  In  the  "As  Is"  process,  what  decisions  are  artifacts  of  the  process,  but  not  essential  to  the 
engineering  development? 

■  Are  there  lost  opportunities  by  making  early  concept  and  design  decisions? 

■  Is  there  a  risk  of  bad  decisions,  risks  and  costs  of  no  or  deferred  decisions,  during  planning,  or 
during  execution? 
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■  Reconsider  the  "full  system"  technical  review  model.  Not  all  parts  of  the  system  are  ready  for 
PDR,  CDR  at  the  same  time.  Some  are  more  mature  than  others.  Maybe  a  granular  approach  is 
needed. 

The  timing  of  technical  reviews  and  decisions  should  be  made  when  there  is  an  accumulation  of 
evidence  sufficient  to  make  a  credible  decision.  Ideally,  this  will  be  inherent  in  the  Vision  concept,  when 
the  required  information  and  associated  analyses  are  complete,  the  evidence  and  timing  for  decisions 
should  be  triggered  events  in  the  automated  workflow. 


5.5  Risk  Related  Research 

SERC  research  teams  are  involved  in  several  related  research  efforts  that  will  be  leveraged  in  the  risk 
framework.  We  need  to  explore  how  the  following  can  be  leveraged: 

■  The  High  Performance  Computing  Modernization  (HPCM)  CREATE  program  to  use  high-fidelity 
models  in  systems  design  is  establishing  a  working  group  on  Uncertainty  Quantification.  SERC 
partners  are  collaborating  with  NAVSEA  and  the  HPCM  program. 

■  The  DARPA  internet-fabrication  (iFab)  project  sponsored  research  by  a  SERC  collaborator  to 
develop  software  to  automatically  detect  and  complete  gaps  in  specifications  for  a  "build  to" 
design. 

■  The  US  Army  TARDEC  is  developing  knowledge  models  to  capture  design  factors  and 
relationships  in  system  design  and  development.  The  resulting  decision  breakdown  structure 
and  process  should  help  distinguish  substantive  design  and  engineering  decisions  versus 
artifacts  of  the  "As  Is"  process.  SERC  partners  are  coordinating  with  this  effort. 

■  OSD  is  sponsoring  a  SERC  project  in  "risk  leading  indicators"  and  "risk  estimating  relationships," 
analyzing  the  consistency,  completeness,  and  complexity  of  the  system  architecture, 
requirements,  task  structure,  and  team  organization,  and  combining  those  with  TRL/IRL  levels 
and  Advancement  Degree  of  Difficulty  indicators  (this  project  is  being  conducted  in 
collaboration  with  TARDEC  and  an  acquisition  program). 

■  The  Engineered  Resilient  Systems  (ERS)  effort  is  addressing  lost  opportunity  by  making  early 
concept  &  design  decisions,  the  time  and  cost  to  reverse  decisions,  and  tradeoffs  between 
timely  but  bad  decisions  versus  deferred  decisions.  SERC  partners  are  collaborating  with  the 
NAVSEA  ERS  and  set-based  design  projects. 


5.6  Risk  OF  SE  Transformation  TO  MBSE 

There  is  also  a  concern  that  the  risk  of  SE  transformation  to  MBSE  will  fail  to  provide  an  efficient, 
effective  and  reliable  alternative  to  the  current  process.  There  are  currently  no  good  ways  to  assess  if  a 
new  MBSE  approach  produces  outcomes  as  good  or  better  than  the  current  process.  Regardless,  it  will 
be  important  to  establish  Measures  of  Effectiveness  (MOEs)  or  Measures  of  Performance  (MOPs). 
However,  this  effort  still  remains  focused  on  assessing  the  technical  feasibility,  and  not  the  adoption  of 
MBSE. 
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6  Conclusion  and  Next  Steps 


This  research  supports  the  SERC's  research  thrust  for  SE  Transformation,  where  the  goai  is  to  move 
from  engineering  approaches  for  systems  designed  for  optimai  performance  against  a  static  set  of 
requirements  over  iong  procurement  cycies  to  approaches  that  enhance  the  productivity  of  engineers 
to  rapidiy  deveiop  cost  effective,  fiexibie,  agiie  systems  that  can  respond  to  evoiving  threats  and  mission 
needs. 

in  our  preiiminary  discussions  with  organizations  in  assessing  the  state-of-the-art  in  MBSE,  we  see  a 
generai  trend  towards  more  widespread  adoption  of  MBSE,  rather  than  the  radicai  transformation 
desired  of  the  NAVAiR  vision.  Our  measurement  instrument  wiii  continue  to  be  refined  in  order  to 
characterize  those  types  of  enabiing  technoiogies,  methods  and  innovations  that  resuit  in  such  a  radicai 
transformation. 

As  we  move  to  Phase  ii,  we  are  iooking  for  existence  proofs  in  exampie  uses  of  MBSE.  We  wiii  deveiop 
case  studies  as  we  focus  on  the  objective  for  Tasks  3  and  Task  4  in  order  to  deveiop  a  modest 
demonstration.  We  wiii  continue  to  deveiop  an  approach  to  map  the  "As  is"  process  to  the  Vision, 
which  shouid  heip  provide  confidence  about  the  compieteness  of  the  Vision  and  ensure  that  we  deai 
with  the  criticai  requirements  of  safety  and  airworthiness. 

The  representation  for  the  Vision  is  more  of  a  reference  architecture  of  the  possibie  types  of  program 
data  that  are  captured;  there  wiii  be  a  need  to  have  dependencies/constraints  as  different  types  of  data 
are  captured  and  needed  depending  for  exampie  based  on:  iand-based,  sea-based,  etc.  We  wiii  refine 
the  Vision  and  iink/map  to  existing  documentation,  untii  the  attributes  associated  with  the  data  are 
formaiized  in  the  Vision  modei.  We  wiii  use  case  study  data  to  scope  the  probiem,  use  and  mine 
historicai  data  from  programs  of  record.  This  shouid  inciude  how  to  capture  and  represent  other 
aspects  such  as  airworthiness  constraints,  affordabiiity  data,  earned  vaiue  data,  and  risk  exampies. 
Many  key  distinguishing  characteristics  of  the  vision  wiii  revoive  around  more  integration, 
interoperabiiity  and  interaction  with  the  modeiing  and  simuiation  worid.  We  wiii  represent  where  and 
how  modeis  are  used  to  produce  modei  artifacts  that  were  typicaiiy  captured  and  reviewed  as 
documents. 

We  are  working  with  our  NAVAiR  consuitants  who  are  characterizing  what  a  "Contractor"  couid  use  as 
modei-based  requirements  and  design  constraints  with  respect  to  this  Vision  concept.  This  inciudes 
what  wouid  be  used  at  a  Digitai  CDR  in  order  to  provide  the  needed  information  and  evidence  for  the 
decision  gates  represented  as  modeis  or  digitai  artifacts  in  order  to  support  risk-based  signoff. 

We  are  stiii  in  need  of  case  study  data,  and  we  wouid  iike  to  better  understand  the  inventory  of 
simuiation  assets  used  by  NAVAiR.  We  wiii  use  case  study  data  to  iook  at  the  various  gates  from  a 
historicai  point  of  view  to  understand  the  data  required  for  decisions  represented  in  the  formai  of 
digitai  evidence. 

The  specific  efforts  for  Phase  ii  wiii  inciude: 

■  Continue  to  conduct  and  anaiyze  the  resuits  from  discussions  with  Government,  industry  and 
Academia,  aiong  with  refinement  of  the  coiiection  and  measurement  instruments 

■  Evoiution  of  the  modei  iexicon,  and  continuous  updates  to  a  website 

■  Modei  of  the  "As  is"  and  mapping  to  modei  of  the  "Vision"  with  traceabiiity  and  compieteness 
anaiysis  to  the  DoD  5000.02 
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■  Risk  management  framework  that  aligns  with  the  'Vision";  this  can  include  examples  such  as  the 
Airworthiness  risk  model,  risk  factor  analysis  from  case  study  data  and/or  program  execution 
risk  models 

■  Demonstration  thread(s)  based  on  case  study  or  surrogate  data;  we  will  either  use  case  study 
data  provided  to  us,  or  we  will  create  some  type  of  surrogate  case  study 


49 


Factor  Definitions 


The  following  is  the  current  set  (fifth  version)  of  the  set  of  factors  associated  with  the  discussion 
measurement  instrument  (see  Section  2).  As  the  discussions  with  organizations  are  held,  these  factors 
and  the  associated  categories  will  be  refined. 

Table  4.  Discussion  Instrument  Factor  Definition 


Factor  Category 

Factors 

General 

These  factors  relate  to  the  degree  to 
which  advance  MBSE  provides  a 
holistic  approach  to  SE 

Commentary 

Magnitude  of 
applicability 

over  the 

lifecycle 

Organizational  Scope 

What  is  the  scope  of  the  MBSE  usage? 
Normally,  when  thinking  about 
NAVAIR  systems  the  scope  is  quite 
large  and  involves  large  programs. 
Therefore,  what  organizational  scope 
does  the  MBSE  usages  apply: 
Program,  Project,  an  entire  Business 
Unit,  Platform  (e.g.,  a  type  of  a 
specific  aircraft,  tank,  automobile). 
Department,  or  Site. 

Key  to  all  of  these  questions  is  that  we 
are  looking  for  "Technical  Feasibility  of 
"doing  everything  with  model."  We 
recognize  that  actual  adoption  can  be 
difficult,  and  might  not  make  sense  on 
older  systems.  Therefore  related  to 
this,  it  is  probably  best  to  ensure  that 
the  question  perspectives  come  from  a 
Chief  Engineer,  Chief  Technical  Offer, 
MBSE  Organizational  Specialist  and 
possibly  the  Program  Manager.  To 
carry  this  a  step  further,  it  might  also 
be  important  to  keep  the  "systems" 
perspective  in  mind,  because  some  of 
the  concepts  discussed  may  have  been 
applied  in  Hardware  and  possibly  in 
Software  (e.g.,  the  control  laws  for  the 
F35  are  built  in  Simulink,  with  auto 
code  generation,  and  a  significant 
portion  of  auto  test  generation),  but 
not  completely  at  the  Systems  level. 

Scope  Impact 

How  broadly  does  the  answers  cover 
the  entire  lifecycle  (for  example,  a 
university  research  project  might  be 
very  advanced  in  terms  of  analysis  or 
simulation,  but  it  does  not  cover  the 
entire  DoD  5000  lifecycle). 

The  answer  to  this  question  has  a  lot  of 
weight,  because  we  need  to  consider 
answer  in  context  of  lifecycle 
applicable  to  NAVAIR  (and  in  general 
DoD  Acq.  Programs). 

Proven  beyond 
a  research 
concept 

Demonstrations 

Are  the  capabilities  discussed  actually 
in  operations  -  have  they  been 
demonstrated? 

We  want  to  understand  that  things 
discussed  are  more  than  just  research 
concepts. 

Crossing  the 
Virtual  V 

Integrated 

Simulation 

In  order  to  Cross  the  Virtual  V,  there 
will  be  many  types  of  modeling  and 
simulation  required  to  support 
various  type  of  domains  within  the 
system.  To  what  degree  are  the 
simulations  integrated,  and  better  yet 
do  different  simulations  work  off  of 

shared  models? 

In  order  to  "cross  the  virtual  V"  during 
the  early  stages  of  development,  it  is 
important  to  understand  if  the 
inputs/outputs  from  one  set  of 
simulations  can  feed  another  (e.g.,  to 
be  able  to  understand  the  capability  in 
the  mission  context) 

Formal  Analysis 

Are  the  analyses  (e.g.,  property 
analysis)  formal,  meaning  that  they 
are  performed  on  models 

automatically? 

Is  the  analysis  fully  automated  from 
the  models  (H)  or  is  there  human 
interpretation  required  (M  or  L)  or 
none  (L)? 
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Domain  Specific 

Are  the  different  types  of  models 
related  to  the  domain?  For  example, 
control  system  engineers  often  use 
Simulink/Matlab.  Also,  most  modeling 
and  simulation  environments  are 
domain-specific. 

Domain-specific  modeling  languages 
are  an  emerging  trend;  these  types  of 
approaches  provide  intuitive 

abstractions  (often  graphical)  that  are 
familiar  to  engineers  within  the 
domain.  Rather,  SysML,  while  good  for 
systems  engineers,  it  might  not  be 
applicable  to  flight  controls,  networks, 
fluid  dynamics,  etc.  In  addition,  there  is 
not  significant  support  for  automated 
V&V  from  SysML  as  the  semantics  are 
not  very  rich. 

Cross  Domain 
Coverage 

Domain 

Interoperability 

Are  the  models  that  are  in  different, 
but  related  domains  integrated?  Are 
the  models  consistent  across  the 

domains? 

For  example,  are  the  models  that  are 
used  for  performance  the  same  models 
used  for  integrity/dependability 

analysis? 

Synthesis/Generation 

Can  the  models  be  used  for 
synthesis/generation  of  other  related 
artifacts  such  as  code,  simulation, 
analysis,  tests  and  documentation 

Meta-Model/Model 

Transformations 

Are  the  models  used  in  one  domain, 
or  for  one  purpose,  transformable 
into  another  domain  where  the  well- 

defined  semantics  in  one  domain  is 
carried  through  the  transformation 
into  the  other  domain;  if  so  are  they 
known  to  be  consistent? 

We  know  that  one  type  of  modeling  is 
not  always  appropriate  for  everything, 
and  that  is  why  there  is  emergence  of 
Domain-Specific  Modeling  languages; 
the  key  question  is:  are  the  models  for 
one  use  consistent  for  other  users 
(e.g.,  performance,  integrity). 

Virtual  System 
Representation 

Surrogate  Integration 

Are  surrogates  used  to  support 
analysis,  and  are  the  results  of  the 
surrogates  captured  so  that  they  can 
be  factored  into  modeling  and 
simulation  in  the  future? 

Example,  Formula  1  racing,  uses  data 
logging  during  physical 

experimentation  and  then  factors 
results  (and  logs)  back  into  simulation 
environment;  can  we  fly  some  new 
capability  on  an  existing  aircraft  and 
then  factor  the  results  from  the  test 
flights  back  into  the  modeling  and 
simulation  environments?  This  in  the 
future  should  allow  more  virtual  flight 
testing  (once  the  approach  becomes 
trusted). 

Formal  Capability 
Assessment 

How  well  do  the  models,  simulations 
and  analyses  capabilities  support  the 
ability  to  understand  the  capabilities 
being  developed? 

Are  the  abstractions  from  the  models 
still  "rich  enough"  to  be  representative 
of  the  actual  mission  environment 

when  used  in  a  virtual  environment? 
For  example,  if  we  use  a  3D  immersive 
environment,  can  we  understand  the 
physical  characteristic  of  the 

operational  system? 

Virtual 

Accuracy/Margin 

Analysis 

Are  the  modeling,  simulation  and 
analysis  accurate?  How  well  do  they 
allow  the  designers  to  understand  the 
margins? 

As  an  example,  margin  tolerances  at 
the  component  level  can  propagate  as 
the  system  is  composed  (or 
assembled).  Are  these  factors 
understood  and  controlled? 

3D  Immersive 

Environments 

What  is  the  degree  to  which  3D 
Immersive  Environments  are  used  to 
improve  the  understanding  (and 
possibly  training)  of  the  virtual 
systems. 
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Management 
Criticality  Risks 
(Relates  to  Task 

4) 

Risk  Management 

Is  there  proper  risk  management 
identification,  analysis  and 

mitigations  applied  based  on  the  use 
of  models? 

This  should  also  consider: 

-  Adequately  deal  with  critical 
timelines 

Integrated  operational  risk 

-  Change  management  (model-based 
change  management  is  different  than 
document-based) 

Predictive  Analytics 

Are  there  models  used  to  support  a 
quantitative  approach  to  risk 
management? 

Model-based  metrics 

Are  there  model-based  metrics  (or  a 
comprehensive  set  of  model 
measurements)  and  are  they  used  to 
support  the  management  of 
programs/projects? 

The  use  of  model-based  metrics 
reflects  on  the  modeling  maturing  of 
the  organization. 

Attributes  of 
Modeling 
Maturity 

Multi-model 
interdependencies  / 
consistency  and 
semantic  precision 

If  the  organization  is  dealing  with 
many  different  types  of  models,  are 
the  interdependencies  managed  and 
are  the  models  semantically  precise 
enough  to  manage  model 

consistency? 

Dealing  with  interdependencies  and 
modeling  consistency  often  deals  with 
having  a  detailed  understanding  of  the 
semantics  across  models.  Positive 
results  for  this  answer  suggest  a  very 
advanced  use  of  models. 

High  Performance 
Computing  (HPC) 

Is  HPC  applied  to  the  modeling, 
simulation  and  analysis  efforts? 

Use  of  HPC  is  an  indicator  of  high 
modeling  maturity. 

Procedures 

Are  the  procedures  for  using  the 
models  understood,  so  that  we  can 
trust  the  model  outputs  to  support 
other  related  types  of  analysis,  both 
in  terms  of  technical  as  well  as  risk? 

This  applies  heavily  in  airworthiness 
(e.g..  Mil  Std.  516) 

Operational 
Risks  (Relates 
to  Task  4) 

Staff  and  Training 

With  the  advances  in  the  technologies 
associated  with  models,  are  the  staff 
and  training  in  place  to  support  the 
use  of  models? 

This  is  another  indicator  of  a  more 
advanced  organization.  As  a  side  effect 
the  use  of  3D  Immersive  systems  can 
be  valuable  in  both  collaboration  and 
early  training. 

Human  Factors 

How  well  are  human  factors 
supported  by  the  modeling, 

simulation  and  analysis  capabilities? 
This  should  consider  Usability. 

Certification 

How  well  do  the  models-based 
automation  and  practices  support 
certifications  (if  required)? 

If  not  applicable  use  M  -  for  Medium 

Indirect  support 
from  Models 

Regulation 

How  well  do  the  models-based 
automation  and  practices  support 
regulations  (if  required)? 

If  not  applicable  use  M  -  for  Medium 

Modeling  and 
Simulation 

Qualification 

How  much  do  we  trust  our  models? 
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Acronyms  and  Abbreviation 


This  section  provides  a  list  of  some  of  the  terms  used  throughout  the  paper.  The  model  lexicon  should 
have  all  of  these  terms  and  many  others. 


AADL 

ACAT 

AGM 

AP233 

ATL 

ASR 

BDD 

BNF 

BPML 

CAD 

CASE 

CDR 

CMM 

CMMI 

CORBA 

CREATE 

CWM 

dB 

DBMS 

DAG 

DARPA 

DAU 

DoD 

DoDAF 

DSL 

DSM 

DSML 

ERS 

HPCM 

IBM 

IBD 

ICD 

INCOSE 

IPR 

IRL 

ISO 

IT 

Linux 

MARTE 

MATRIXx 

MBT 

MDA® 

MDD™ 


Architecture  Analysis  &  Design  Language 
Acquisition  Category 
Acquisition  Guidance  Model 
Application  Protocol  233 
ATLAS  Transformation  Language 
Alternative  System  Review 
SysML  Block  Definition  Diagram 
Backus  Naur  Form 

Business  Process  Modeling  Language 
Computer-Aided  Design 
Computer-Aided  Software  Engineering 
Critical  Design  Review 
Capability  Maturity  Model 
Capability  Maturity  Model  Integration 
Common  Object  Requesting  Broker  Architecture 

Computational  Research  and  Engineering  for  Acquisition  Tools  and  Environments 

Common  Warehouse  Metamodel 

Decibel 

Database  Management  System 
Defense  Acquisition  Guidebook 
Defense  Advanced  Research  Project  Agency 
Defense  Acquisition  University 
Department  of  Defense 

Department  of  Defense  Architectural  Framework 

Domain  Specific  Languages 

Domain  Specific  Modeling 

Domain  Specific  Modeling  Language 

Engineered  Resilient  Systems 

High  Performance  Computing  Modernization 

International  Business  Machines 

SysML  Internal  Block  Diagram 

Interface  Control  Document 

International  Council  on  Systems  Engineering 

Integration  Problem  Report 

Integration  Readiness  Level 

International  Organization  for  Standardization 

Information  Technology 

An  operating  system  created  by  Linus  Torvalds 

Modeling  and  Analysis  of  Real  Time  Embedded  systems 

Product  family  for  model-based  control  system  design  produced  by  National 

Instruments 

Model  Based  Testing 

Model  Driven  Architecture® 

Model  Driven  Development 
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MDE 

Model  Driven  Engineering 

MDSD 

Model  Driven  Software  Development 

MDSE 

Model  Driven  Software  Engineering 

MIC 

Model  Integrated  Computing 

MMM 

Modeling  Maturity  Model 

MoDAF 

United  Kingdom  Ministry  of  Defence  Architectural  Framework 

MOE 

Measure  of  Effectiveness 

MOF 

Meta  Object  Facility 

MOP 

Measure  of  Performance 

MVS 

Multiple  Virtual  Storage 

NASA 

National  Aeronautics  and  Space  Administration 

NAVAIR 

U.S.  Navy  Naval  Air  Systems  Command 

NAVSEA 

U.S.  Naval  Sea  Systems  Command 

OCL 

Object  Constraint  Language 

OMG 

Object  Management  Group 

00 

Object  oriented 

OSD 

Office  of  the  Secretary  of  Defense 

PDM 

Product  Data  Management 

PDR 

Preliminary  Design  Review 

PIM 

Platform  Independent  Model 

PLM 

Product  Lifecycle  Management 

POR 

Program  of  Record 

PRR 

Production  Readiness  Review 

PSM 

Platform  Specific  Model 

RFP 

Request  for  Proposal 

ROI 

Return  On  Investment 

SE 

System  Engineering 

SERC 

Systems  Engineering  Research  Center 

SETR 

System  Engineering  Technical  Review 

Simulink/Stateflow 

Product  family  for  model-based  control  system  produced  by  The  Mathworks 

SCR 

Software  Cost  Reduction 

SDD 

Software  Design  Document 

SFR 

System  Functional  Review 

SOAP 

A  protocol  for  exchanging  XML-based  messages  -  originally  stood  for  Simple  Object 
Access  Protocol 

Software  Factory 

Term  used  by  Microsoft 

SRR 

System  Requirements  Review 

SRS 

Software  Requirement  Specification 

STOVL 

Short  takeoff  and  vertical  landing 

SVR 

System  Verification  Review 

SysML 

System  Modeling  Language 

TARDEC 

US  Army  Tank  Automotive  Research 

TRL 

Technology  Readiness  Level 

TRR 

Test  Readiness  Review 

UML 

Unified  Modeling  Language 

XMI 

XML  Metadata  Interchange 

XML 

extensible  Markup  Language 

US 

United  States 

XSLT 

extensible  Stylesheet  Language  family  (XSL)  Transformation 
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xUML 

Unix 

VHDL 

VxWorks 


Executable  UML 

An  operating  system  with  trademark  held  by  the  Open  Group 
Verilog  Hardware  Description  Language 

Operating  system  designed  for  embedded  systems  and  owned  by  WindRiver 
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Trademarks 


Astah  SysML  is  Copyright  of  Change  Vision,  Inc. 

BridgePoint  is  a  registered  trademark  of  Mentor  Graphics. 

CORE  is  a  registered  trademark  of  Vitech  Corporation. 

is  a  trademark  of  the  IBM  Corporation 
iGrafx  is  a  registered  trademark  of  iGrafx,  LCC. 

Java^"^  and  J2EE^'^  are  trademark  of  SUN  Microsystems 
Java  is  trademarked  by  Sun  Microsystems,  Inc. 

LDRA  is  a  registered  trademark  of  Trademark  of  LDRA  Ltd.  and  Subsidiaries. 

Linux  is  a  registered  trademark  of  Linux  Mark  Institute. 

Mathworks,  Simulink,  and  Stateflow  are  registered  trademarks  of  The  Mathworks,  Inc. 

MagicDraw  is  a  trademark  of  No  Magic,  Inc. 

MATRIXx  is  a  registered  trademark  of  National  Instruments. 

Object  Management  Group  (OMG):  OMG's  Registered  Trademarks  include:  MDA®,  Model  Driven 
Architecture®,  UML®  ,  CORBA®,  CORBA  Academy®,  XMI® 

OMG's  Trademarks  include,  CWM^"^  ,  Model  Based  Application  Development^"^,  MDD^"^,  Model  Based 
Development^"^,  Model  Based  Managements"^,  Model  Based  Programmings"^,  Model  Driven  Application 
Developments"^,  Model  Driven  Developments"^ 

ModelCenter,  is  a  registered  trademark  of  Phoenix  Integration,  Inc. 

Model  Driven  Programmings"^,  Model  Driven  SystemsS"^,  OMG  Interface  Definition  Language  (IDL)s"^, 
Unified  Modeling  Languages"^,  «UML»s"^ 

OMG®,  MDA®,  UML®,  MOF®,  XMI®,  SysMLs^,  BPMLs^  are  registered  trademarks  or  trademarks  of  the 
Object  Management  Group. 

PowerPoint  is  a  registered  trademark  of  Microsoft,  Inc. 

Real-time  Studio  Professional  is  a  registered  trademark  of  ARTiSAN  Software  Tools,  Inc. 

Rhapsody  is  a  registered  trademark  of  Telelogic/IBM. 

Rose  XDE  is  a  registered  trademark  of  IBM. 

SCADE  is  copyrighted  to  Esterel  Technologies. 

Simulink  is  a  registered  trademark  of  The  MathWorks. 

Stateflow  is  a  registered  trademark  of  The  MathWorks. 

Statemate  is  a  registered  trademark  of  Telelogic/IBM. 

UNIX  is  a  registered  trademark  of  The  Open  Group. 

VAPS  is  registered  at  eNGENUITY  Technologies. 

VectorCAST  is  a  registered  trademark  of  Vector  Software,  Inc. 

Visio  is  a  registered  trademark  of  Microsoft,  Inc. 

VxWorks  is  a  registered  trademark  of  Wind  River  Systems,  Inc. 

Windows  is  a  registered  trademark  of  Microsoft  Corporation  in  the  United  States  and  other  countries. 
XML^^  is  a  trademark  of  W3C 

All  other  trademarks  belong  to  their  respective  organizations. 
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